Hugging Face's logo

Transformers documentation

Auto Classes

Hugging Face's logo
Join the Hugging Face community

and get access to the augmented documentation experience

to get started

Auto Classes

多くの場合、from_pretrained()メソッドに与えられた事前学習済みモデルの名前やパスから、使用したいアーキテクチャを推測することができます。自動クラスはこの仕事をあなたに代わって行うためにここにありますので、事前学習済みの重み/設定/語彙への名前/パスを与えると自動的に関連するモデルを取得できます。

AutoConfig、AutoModel、AutoTokenizerのいずれかをインスタンス化すると、関連するアーキテクチャのクラスが直接作成されます。例えば、

model = AutoModel.from_pretrained("google-bert/bert-base-cased")

これはBertModelのインスタンスであるモデルを作成します。

各タスクごと、そして各バックエンド（PyTorch、TensorFlow、またはFlax）ごとにAutoModelのクラスが存在します。

自動クラスの拡張

それぞれの自動クラスには、カスタムクラスで拡張するためのメソッドがあります。例えば、NewModelというモデルのカスタムクラスを定義した場合、NewModelConfigを確保しておけばこのようにして自動クラスに追加することができます：

from transformers import AutoConfig, AutoModel

AutoConfig.register("new-model", NewModelConfig)
AutoModel.register(NewModelConfig, NewModel)

その後、通常どおりauto classesを使用することができるようになります！

あなたのNewModelConfigがPretrainedConfigのサブクラスである場合、そのmodel_type属性がコンフィグを登録するときに使用するキー（ここでは"new-model"）と同じに設定されていることを確認してください。

同様に、あなたのNewModelがPreTrainedModelのサブクラスである場合、そのconfig_class属性がモデルを登録する際に使用するクラス（ここではNewModelConfig）と同じに設定されていることを確認してください。

AutoConfig

class transformers.AutoConfig

< >

( )

This is a generic configuration class that will be instantiated as one of the configuration classes of the library when created with the from_pretrained() class method.

This class cannot be instantiated directly using __init__() (throws an error).

from_pretrained

< >

( pretrained_model_name_or_path **kwargs )

Parameters

  • pretrained_model_name_or_path (str or os.PathLike) — Can be either:

    • A string, the model id of a pretrained model configuration hosted inside a model repo on huggingface.co.
    • A path to a directory containing a configuration file saved using the save_pretrained() method, or the save_pretrained() method, e.g., ./my_model_directory/.
    • A path or url to a saved configuration JSON file, e.g., ./my_model_directory/configuration.json.
  • cache_dir (str or os.PathLike, optional) — Path to a directory in which a downloaded pretrained model configuration should be cached if the standard cache should not be used.
  • force_download (bool, optional, defaults to False) — Whether or not to force the (re-)download the model weights and configuration files and override the cached versions if they exist.
  • resume_download (bool, optional, defaults to False) — Whether or not to delete incompletely received files. Will attempt to resume the download if such a file exists.
  • proxies (Dict[str, str], optional) — A dictionary of proxy servers to use by protocol or endpoint, e.g., {'http': 'foo.bar:3128', 'http://hostname': 'foo.bar:4012'}. The proxies are used on each request.
  • revision (str, optional, defaults to "main") — The specific model version to use. It can be a branch name, a tag name, or a commit id, since we use a git-based system for storing models and other artifacts on huggingface.co, so revision can be any identifier allowed by git.
  • return_unused_kwargs (bool, optional, defaults to False) — If False, then this function returns just the final configuration object.

    If True, then this functions returns a Tuple(config, unused_kwargs) where unused_kwargs is a dictionary consisting of the key/value pairs whose keys are not configuration attributes: i.e., the part of kwargs which has not been used to update config and is otherwise ignored.

  • trust_remote_code (bool, optional, defaults to False) — Whether or not to allow for custom models defined on the Hub in their own modeling files. This option should only be set to True for repositories you trust and in which you have read the code, as it will execute code present on the Hub on your local machine.
  • kwargs(additional keyword arguments, optional) — The values in kwargs of any keys which are configuration attributes will be used to override the loaded values. Behavior concerning key/value pairs whose keys are not configuration attributes is controlled by the return_unused_kwargs keyword parameter.

Instantiate one of the configuration classes of the library from a pretrained model configuration.

The configuration class to instantiate is selected based on the model_type property of the config object that is loaded, or when it’s missing, by falling back to using pattern matching on pretrained_model_name_or_path:

  • albert — AlbertConfig (ALBERT model)
  • align — AlignConfig (ALIGN model)
  • altclip — AltCLIPConfig (AltCLIP model)
  • audio-spectrogram-transformer — ASTConfig (Audio Spectrogram Transformer model)
  • autoformer — AutoformerConfig (Autoformer model)
  • bark — BarkConfig (Bark model)
  • bart — BartConfig (BART model)
  • beit — BeitConfig (BEiT model)
  • bert — BertConfig (BERT model)
  • bert-generation — BertGenerationConfig (Bert Generation model)
  • big_bird — BigBirdConfig (BigBird model)
  • bigbird_pegasus — BigBirdPegasusConfig (BigBird-Pegasus model)
  • biogpt — BioGptConfig (BioGpt model)
  • bit — BitConfig (BiT model)
  • blenderbot — BlenderbotConfig (Blenderbot model)
  • blenderbot-small — BlenderbotSmallConfig (BlenderbotSmall model)
  • blip — BlipConfig (BLIP model)
  • blip-2 — Blip2Config (BLIP-2 model)
  • bloom — BloomConfig (BLOOM model)
  • bridgetower — BridgeTowerConfig (BridgeTower model)
  • bros — BrosConfig (BROS model)
  • camembert — CamembertConfig (CamemBERT model)
  • canine — CanineConfig (CANINE model)
  • chinese_clip — ChineseCLIPConfig (Chinese-CLIP model)
  • chinese_clip_vision_model — ChineseCLIPVisionConfig (ChineseCLIPVisionModel model)
  • clap — ClapConfig (CLAP model)
  • clip — CLIPConfig (CLIP model)
  • clip_vision_model — CLIPVisionConfig (CLIPVisionModel model)
  • clipseg — CLIPSegConfig (CLIPSeg model)
  • clvp — ClvpConfig (CLVP model)
  • code_llama — LlamaConfig (CodeLlama model)
  • codegen — CodeGenConfig (CodeGen model)
  • cohere — CohereConfig (Cohere model)
  • conditional_detr — ConditionalDetrConfig (Conditional DETR model)
  • convbert — ConvBertConfig (ConvBERT model)
  • convnext — ConvNextConfig (ConvNeXT model)
  • convnextv2 — ConvNextV2Config (ConvNeXTV2 model)
  • cpmant — CpmAntConfig (CPM-Ant model)
  • ctrl — CTRLConfig (CTRL model)
  • cvt — CvtConfig (CvT model)
  • data2vec-audio — Data2VecAudioConfig (Data2VecAudio model)
  • data2vec-text — Data2VecTextConfig (Data2VecText model)
  • data2vec-vision — Data2VecVisionConfig (Data2VecVision model)
  • dbrx — DbrxConfig (DBRX model)
  • deberta — DebertaConfig (DeBERTa model)
  • deberta-v2 — DebertaV2Config (DeBERTa-v2 model)
  • decision_transformer — DecisionTransformerConfig (Decision Transformer model)
  • deformable_detr — DeformableDetrConfig (Deformable DETR model)
  • deit — DeiTConfig (DeiT model)
  • depth_anything — DepthAnythingConfig (Depth Anything model)
  • deta — DetaConfig (DETA model)
  • detr — DetrConfig (DETR model)
  • dinat — DinatConfig (DiNAT model)
  • dinov2 — Dinov2Config (DINOv2 model)
  • distilbert — DistilBertConfig (DistilBERT model)
  • donut-swin — DonutSwinConfig (DonutSwin model)
  • dpr — DPRConfig (DPR model)
  • dpt — DPTConfig (DPT model)
  • efficientformer — EfficientFormerConfig (EfficientFormer model)
  • efficientnet — EfficientNetConfig (EfficientNet model)
  • electra — ElectraConfig (ELECTRA model)
  • encodec — EncodecConfig (EnCodec model)
  • encoder-decoder — EncoderDecoderConfig (Encoder decoder model)
  • ernie — ErnieConfig (ERNIE model)
  • ernie_m — ErnieMConfig (ErnieM model)
  • esm — EsmConfig (ESM model)
  • falcon — FalconConfig (Falcon model)
  • fastspeech2_conformer — FastSpeech2ConformerConfig (FastSpeech2Conformer model)
  • flaubert — FlaubertConfig (FlauBERT model)
  • flava — FlavaConfig (FLAVA model)
  • fnet — FNetConfig (FNet model)
  • focalnet — FocalNetConfig (FocalNet model)
  • fsmt — FSMTConfig (FairSeq Machine-Translation model)
  • funnel — FunnelConfig (Funnel Transformer model)
  • fuyu — FuyuConfig (Fuyu model)
  • gemma — GemmaConfig (Gemma model)
  • git — GitConfig (GIT model)
  • glpn — GLPNConfig (GLPN model)
  • gpt-sw3 — GPT2Config (GPT-Sw3 model)
  • gpt2 — GPT2Config (OpenAI GPT-2 model)
  • gpt_bigcode — GPTBigCodeConfig (GPTBigCode model)
  • gpt_neo — GPTNeoConfig (GPT Neo model)
  • gpt_neox — GPTNeoXConfig (GPT NeoX model)
  • gpt_neox_japanese — GPTNeoXJapaneseConfig (GPT NeoX Japanese model)
  • gptj — GPTJConfig (GPT-J model)
  • gptsan-japanese — GPTSanJapaneseConfig (GPTSAN-japanese model)
  • graphormer — GraphormerConfig (Graphormer model)
  • grounding-dino — GroundingDinoConfig (Grounding DINO model)
  • groupvit — GroupViTConfig (GroupViT model)
  • hubert — HubertConfig (Hubert model)
  • ibert — IBertConfig (I-BERT model)
  • idefics — IdeficsConfig (IDEFICS model)
  • idefics2 — Idefics2Config (Idefics2 model)
  • imagegpt — ImageGPTConfig (ImageGPT model)
  • informer — InformerConfig (Informer model)
  • instructblip — InstructBlipConfig (InstructBLIP model)
  • jamba — JambaConfig (Jamba model)
  • jukebox — JukeboxConfig (Jukebox model)
  • kosmos-2 — Kosmos2Config (KOSMOS-2 model)
  • layoutlm — LayoutLMConfig (LayoutLM model)
  • layoutlmv2 — LayoutLMv2Config (LayoutLMv2 model)
  • layoutlmv3 — LayoutLMv3Config (LayoutLMv3 model)
  • led — LEDConfig (LED model)
  • levit — LevitConfig (LeViT model)
  • lilt — LiltConfig (LiLT model)
  • llama — LlamaConfig (LLaMA model)
  • llava — LlavaConfig (LLaVa model)
  • llava_next — LlavaNextConfig (LLaVA-NeXT model)
  • longformer — LongformerConfig (Longformer model)
  • longt5 — LongT5Config (LongT5 model)
  • luke — LukeConfig (LUKE model)
  • lxmert — LxmertConfig (LXMERT model)
  • m2m_100 — M2M100Config (M2M100 model)
  • mamba — MambaConfig (Mamba model)
  • marian — MarianConfig (Marian model)
  • markuplm — MarkupLMConfig (MarkupLM model)
  • mask2former — Mask2FormerConfig (Mask2Former model)
  • maskformer — MaskFormerConfig (MaskFormer model)
  • maskformer-swin — MaskFormerSwinConfig (MaskFormerSwin model)
  • mbart — MBartConfig (mBART model)
  • mctct — MCTCTConfig (M-CTC-T model)
  • mega — MegaConfig (MEGA model)
  • megatron-bert — MegatronBertConfig (Megatron-BERT model)
  • mgp-str — MgpstrConfig (MGP-STR model)
  • mistral — MistralConfig (Mistral model)
  • mixtral — MixtralConfig (Mixtral model)
  • mobilebert — MobileBertConfig (MobileBERT model)
  • mobilenet_v1 — MobileNetV1Config (MobileNetV1 model)
  • mobilenet_v2 — MobileNetV2Config (MobileNetV2 model)
  • mobilevit — MobileViTConfig (MobileViT model)
  • mobilevitv2 — MobileViTV2Config (MobileViTV2 model)
  • mpnet — MPNetConfig (MPNet model)
  • mpt — MptConfig (MPT model)
  • mra — MraConfig (MRA model)
  • mt5 — MT5Config (MT5 model)
  • musicgen — MusicgenConfig (MusicGen model)
  • musicgen_melody — MusicgenMelodyConfig (MusicGen Melody model)
  • mvp — MvpConfig (MVP model)
  • nat — NatConfig (NAT model)
  • nezha — NezhaConfig (Nezha model)
  • nllb-moe — NllbMoeConfig (NLLB-MOE model)
  • nougat — VisionEncoderDecoderConfig (Nougat model)
  • nystromformer — NystromformerConfig (Nyströmformer model)
  • olmo — OlmoConfig (OLMo model)
  • oneformer — OneFormerConfig (OneFormer model)
  • open-llama — OpenLlamaConfig (OpenLlama model)
  • openai-gpt — OpenAIGPTConfig (OpenAI GPT model)
  • opt — OPTConfig (OPT model)
  • owlv2 — Owlv2Config (OWLv2 model)
  • owlvit — OwlViTConfig (OWL-ViT model)
  • patchtsmixer — PatchTSMixerConfig (PatchTSMixer model)
  • patchtst — PatchTSTConfig (PatchTST model)
  • pegasus — PegasusConfig (Pegasus model)
  • pegasus_x — PegasusXConfig (PEGASUS-X model)
  • perceiver — PerceiverConfig (Perceiver model)
  • persimmon — PersimmonConfig (Persimmon model)
  • phi — PhiConfig (Phi model)
  • pix2struct — Pix2StructConfig (Pix2Struct model)
  • plbart — PLBartConfig (PLBart model)
  • poolformer — PoolFormerConfig (PoolFormer model)
  • pop2piano — Pop2PianoConfig (Pop2Piano model)
  • prophetnet — ProphetNetConfig (ProphetNet model)
  • pvt — PvtConfig (PVT model)
  • pvt_v2 — PvtV2Config (PVTv2 model)
  • qdqbert — QDQBertConfig (QDQBert model)
  • qwen2 — Qwen2Config (Qwen2 model)
  • qwen2_moe — Qwen2MoeConfig (Qwen2MoE model)
  • rag — RagConfig (RAG model)
  • realm — RealmConfig (REALM model)
  • recurrent_gemma — RecurrentGemmaConfig (RecurrentGemma model)
  • reformer — ReformerConfig (Reformer model)
  • regnet — RegNetConfig (RegNet model)
  • rembert — RemBertConfig (RemBERT model)
  • resnet — ResNetConfig (ResNet model)
  • retribert — RetriBertConfig (RetriBERT model)
  • roberta — RobertaConfig (RoBERTa model)
  • roberta-prelayernorm — RobertaPreLayerNormConfig (RoBERTa-PreLayerNorm model)
  • roc_bert — RoCBertConfig (RoCBert model)
  • roformer — RoFormerConfig (RoFormer model)
  • rwkv — RwkvConfig (RWKV model)
  • sam — SamConfig (SAM model)
  • seamless_m4t — SeamlessM4TConfig (SeamlessM4T model)
  • seamless_m4t_v2 — SeamlessM4Tv2Config (SeamlessM4Tv2 model)
  • segformer — SegformerConfig (SegFormer model)
  • seggpt — SegGptConfig (SegGPT model)
  • sew — SEWConfig (SEW model)
  • sew-d — SEWDConfig (SEW-D model)
  • siglip — SiglipConfig (SigLIP model)
  • siglip_vision_model — SiglipVisionConfig (SiglipVisionModel model)
  • speech-encoder-decoder — SpeechEncoderDecoderConfig (Speech Encoder decoder model)
  • speech_to_text — Speech2TextConfig (Speech2Text model)
  • speech_to_text_2 — Speech2Text2Config (Speech2Text2 model)
  • speecht5 — SpeechT5Config (SpeechT5 model)
  • splinter — SplinterConfig (Splinter model)
  • squeezebert — SqueezeBertConfig (SqueezeBERT model)
  • stablelm — StableLmConfig (StableLm model)
  • starcoder2 — Starcoder2Config (Starcoder2 model)
  • superpoint — SuperPointConfig (SuperPoint model)
  • swiftformer — SwiftFormerConfig (SwiftFormer model)
  • swin — SwinConfig (Swin Transformer model)
  • swin2sr — Swin2SRConfig (Swin2SR model)
  • swinv2 — Swinv2Config (Swin Transformer V2 model)
  • switch_transformers — SwitchTransformersConfig (SwitchTransformers model)
  • t5 — T5Config (T5 model)
  • table-transformer — TableTransformerConfig (Table Transformer model)
  • tapas — TapasConfig (TAPAS model)
  • time_series_transformer — TimeSeriesTransformerConfig (Time Series Transformer model)
  • timesformer — TimesformerConfig (TimeSformer model)
  • timm_backbone — TimmBackboneConfig (TimmBackbone model)
  • trajectory_transformer — TrajectoryTransformerConfig (Trajectory Transformer model)
  • transfo-xl — TransfoXLConfig (Transformer-XL model)
  • trocr — TrOCRConfig (TrOCR model)
  • tvlt — TvltConfig (TVLT model)
  • tvp — TvpConfig (TVP model)
  • udop — UdopConfig (UDOP model)
  • umt5 — UMT5Config (UMT5 model)
  • unispeech — UniSpeechConfig (UniSpeech model)
  • unispeech-sat — UniSpeechSatConfig (UniSpeechSat model)
  • univnet — UnivNetConfig (UnivNet model)
  • upernet — UperNetConfig (UPerNet model)
  • van — VanConfig (VAN model)
  • videomae — VideoMAEConfig (VideoMAE model)
  • vilt — ViltConfig (ViLT model)
  • vipllava — VipLlavaConfig (VipLlava model)
  • vision-encoder-decoder — VisionEncoderDecoderConfig (Vision Encoder decoder model)
  • vision-text-dual-encoder — VisionTextDualEncoderConfig (VisionTextDualEncoder model)
  • visual_bert — VisualBertConfig (VisualBERT model)
  • vit — ViTConfig (ViT model)
  • vit_hybrid — ViTHybridConfig (ViT Hybrid model)
  • vit_mae — ViTMAEConfig (ViTMAE model)
  • vit_msn — ViTMSNConfig (ViTMSN model)
  • vitdet — VitDetConfig (VitDet model)
  • vitmatte — VitMatteConfig (ViTMatte model)
  • vits — VitsConfig (VITS model)
  • vivit — VivitConfig (ViViT model)
  • wav2vec2 — Wav2Vec2Config (Wav2Vec2 model)
  • wav2vec2-bert — Wav2Vec2BertConfig (Wav2Vec2-BERT model)
  • wav2vec2-conformer — Wav2Vec2ConformerConfig (Wav2Vec2-Conformer model)
  • wavlm — WavLMConfig (WavLM model)
  • whisper — WhisperConfig (Whisper model)
  • xclip — XCLIPConfig (X-CLIP model)
  • xglm — XGLMConfig (XGLM model)
  • xlm — XLMConfig (XLM model)
  • xlm-prophetnet — XLMProphetNetConfig (XLM-ProphetNet model)
  • xlm-roberta — XLMRobertaConfig (XLM-RoBERTa model)
  • xlm-roberta-xl — XLMRobertaXLConfig (XLM-RoBERTa-XL model)
  • xlnet — XLNetConfig (XLNet model)
  • xmod — XmodConfig (X-MOD model)
  • yolos — YolosConfig (YOLOS model)
  • yoso — YosoConfig (YOSO model)

Examples:

>>> from transformers import AutoConfig

>>> # Download configuration from huggingface.co and cache.
>>> config = AutoConfig.from_pretrained("google-bert/bert-base-uncased")

>>> # Download configuration from huggingface.co (user-uploaded) and cache.
>>> config = AutoConfig.from_pretrained("dbmdz/bert-base-german-cased")

>>> # If configuration file is in a directory (e.g., was saved using *save_pretrained('./test/saved_model/')*).
>>> config = AutoConfig.from_pretrained("./test/bert_saved_model/")

>>> # Load a specific configuration file.
>>> config = AutoConfig.from_pretrained("./test/bert_saved_model/my_configuration.json")

>>> # Change some config attributes when loading a pretrained config.
>>> config = AutoConfig.from_pretrained("google-bert/bert-base-uncased", output_attentions=True, foo=False)
>>> config.output_attentions
True

>>> config, unused_kwargs = AutoConfig.from_pretrained(
...     "google-bert/bert-base-uncased", output_attentions=True, foo=False, return_unused_kwargs=True
... )
>>> config.output_attentions
True

>>> unused_kwargs
{'foo': False}

register

< >

( model_type config exist_ok = False )

Parameters

  • model_type (str) — The model type like “bert” or “gpt”.
  • config (PretrainedConfig) — The config to register.

Register a new configuration for this class.

AutoTokenizer

class transformers.AutoTokenizer

< >

( )

This is a generic tokenizer class that will be instantiated as one of the tokenizer classes of the library when created with the AutoTokenizer.from_pretrained() class method.

This class cannot be instantiated directly using __init__() (throws an error).

from_pretrained

< >

( pretrained_model_name_or_path *inputs **kwargs )

Parameters

  • pretrained_model_name_or_path (str or os.PathLike) — Can be either:

    • A string, the model id of a predefined tokenizer hosted inside a model repo on huggingface.co.
    • A path to a directory containing vocabulary files required by the tokenizer, for instance saved using the save_pretrained() method, e.g., ./my_model_directory/.
    • A path or url to a single saved vocabulary file if and only if the tokenizer only requires a single vocabulary file (like Bert or XLNet), e.g.: ./my_model_directory/vocab.txt. (Not applicable to all derived classes)
  • inputs (additional positional arguments, optional) — Will be passed along to the Tokenizer __init__() method.
  • config (PretrainedConfig, optional) — The configuration object used to determine the tokenizer class to instantiate.
  • cache_dir (str or os.PathLike, optional) — Path to a directory in which a downloaded pretrained model configuration should be cached if the standard cache should not be used.
  • force_download (bool, optional, defaults to False) — Whether or not to force the (re-)download the model weights and configuration files and override the cached versions if they exist.
  • resume_download (bool, optional, defaults to False) — Whether or not to delete incompletely received files. Will attempt to resume the download if such a file exists.
  • proxies (Dict[str, str], optional) — A dictionary of proxy servers to use by protocol or endpoint, e.g., {'http': 'foo.bar:3128', 'http://hostname': 'foo.bar:4012'}. The proxies are used on each request.
  • revision (str, optional, defaults to "main") — The specific model version to use. It can be a branch name, a tag name, or a commit id, since we use a git-based system for storing models and other artifacts on huggingface.co, so revision can be any identifier allowed by git.
  • subfolder (str, optional) — In case the relevant files are located inside a subfolder of the model repo on huggingface.co (e.g. for facebook/rag-token-base), specify it here.
  • use_fast (bool, optional, defaults to True) — Use a fast Rust-based tokenizer if it is supported for a given model. If a fast tokenizer is not available for a given model, a normal Python-based tokenizer is returned instead.
  • tokenizer_type (str, optional) — Tokenizer type to be loaded.
  • trust_remote_code (bool, optional, defaults to False) — Whether or not to allow for custom models defined on the Hub in their own modeling files. This option should only be set to True for repositories you trust and in which you have read the code, as it will execute code present on the Hub on your local machine.
  • kwargs (additional keyword arguments, optional) — Will be passed to the Tokenizer __init__() method. Can be used to set special tokens like bos_token, eos_token, unk_token, sep_token, pad_token, cls_token, mask_token, additional_special_tokens. See parameters in the __init__() for more details.

Instantiate one of the tokenizer classes of the library from a pretrained model vocabulary.

The tokenizer class to instantiate is selected based on the model_type property of the config object (either passed as an argument or loaded from pretrained_model_name_or_path if possible), or when it’s missing, by falling back to using pattern matching on pretrained_model_name_or_path:

  • albert — AlbertTokenizer or AlbertTokenizerFast (ALBERT model)
  • align — BertTokenizer or BertTokenizerFast (ALIGN model)
  • bark — BertTokenizer or BertTokenizerFast (Bark model)
  • bart — BartTokenizer or BartTokenizerFast (BART model)
  • barthez — BarthezTokenizer or BarthezTokenizerFast (BARThez model)
  • bartpho — BartphoTokenizer (BARTpho model)
  • bert — BertTokenizer or BertTokenizerFast (BERT model)
  • bert-generation — BertGenerationTokenizer (Bert Generation model)
  • bert-japanese — BertJapaneseTokenizer (BertJapanese model)
  • bertweet — BertweetTokenizer (BERTweet model)
  • big_bird — BigBirdTokenizer or BigBirdTokenizerFast (BigBird model)
  • bigbird_pegasus — PegasusTokenizer or PegasusTokenizerFast (BigBird-Pegasus model)
  • biogpt — BioGptTokenizer (BioGpt model)
  • blenderbot — BlenderbotTokenizer or BlenderbotTokenizerFast (Blenderbot model)
  • blenderbot-small — BlenderbotSmallTokenizer (BlenderbotSmall model)
  • blip — BertTokenizer or BertTokenizerFast (BLIP model)
  • blip-2 — GPT2Tokenizer or GPT2TokenizerFast (BLIP-2 model)
  • bloom — BloomTokenizerFast (BLOOM model)
  • bridgetower — RobertaTokenizer or RobertaTokenizerFast (BridgeTower model)
  • bros — BertTokenizer or BertTokenizerFast (BROS model)
  • byt5 — ByT5Tokenizer (ByT5 model)
  • camembert — CamembertTokenizer or CamembertTokenizerFast (CamemBERT model)
  • canine — CanineTokenizer (CANINE model)
  • chinese_clip — BertTokenizer or BertTokenizerFast (Chinese-CLIP model)
  • clap — RobertaTokenizer or RobertaTokenizerFast (CLAP model)
  • clip — CLIPTokenizer or CLIPTokenizerFast (CLIP model)
  • clipseg — CLIPTokenizer or CLIPTokenizerFast (CLIPSeg model)
  • clvp — ClvpTokenizer (CLVP model)
  • code_llama — CodeLlamaTokenizer or CodeLlamaTokenizerFast (CodeLlama model)
  • codegen — CodeGenTokenizer or CodeGenTokenizerFast (CodeGen model)
  • cohere — CohereTokenizerFast (Cohere model)
  • convbert — ConvBertTokenizer or ConvBertTokenizerFast (ConvBERT model)
  • cpm — CpmTokenizer or CpmTokenizerFast (CPM model)
  • cpmant — CpmAntTokenizer (CPM-Ant model)
  • ctrl — CTRLTokenizer (CTRL model)
  • data2vec-audio — Wav2Vec2CTCTokenizer (Data2VecAudio model)
  • data2vec-text — RobertaTokenizer or RobertaTokenizerFast (Data2VecText model)
  • dbrx — GPT2Tokenizer or GPT2TokenizerFast (DBRX model)
  • deberta — DebertaTokenizer or DebertaTokenizerFast (DeBERTa model)
  • deberta-v2 — DebertaV2Tokenizer or DebertaV2TokenizerFast (DeBERTa-v2 model)
  • distilbert — DistilBertTokenizer or DistilBertTokenizerFast (DistilBERT model)
  • dpr — DPRQuestionEncoderTokenizer or DPRQuestionEncoderTokenizerFast (DPR model)
  • electra — ElectraTokenizer or ElectraTokenizerFast (ELECTRA model)
  • ernie — BertTokenizer or BertTokenizerFast (ERNIE model)
  • ernie_m — ErnieMTokenizer (ErnieM model)
  • esm — EsmTokenizer (ESM model)
  • falcon — PreTrainedTokenizerFast (Falcon model)
  • fastspeech2_conformer — (FastSpeech2Conformer model)
  • flaubert — FlaubertTokenizer (FlauBERT model)
  • fnet — FNetTokenizer or FNetTokenizerFast (FNet model)
  • fsmt — FSMTTokenizer (FairSeq Machine-Translation model)
  • funnel — FunnelTokenizer or FunnelTokenizerFast (Funnel Transformer model)
  • gemma — GemmaTokenizer or GemmaTokenizerFast (Gemma model)
  • git — BertTokenizer or BertTokenizerFast (GIT model)
  • gpt-sw3 — GPTSw3Tokenizer (GPT-Sw3 model)
  • gpt2 — GPT2Tokenizer or GPT2TokenizerFast (OpenAI GPT-2 model)
  • gpt_bigcode — GPT2Tokenizer or GPT2TokenizerFast (GPTBigCode model)
  • gpt_neo — GPT2Tokenizer or GPT2TokenizerFast (GPT Neo model)
  • gpt_neox — GPTNeoXTokenizerFast (GPT NeoX model)
  • gpt_neox_japanese — GPTNeoXJapaneseTokenizer (GPT NeoX Japanese model)
  • gptj — GPT2Tokenizer or GPT2TokenizerFast (GPT-J model)
  • gptsan-japanese — GPTSanJapaneseTokenizer (GPTSAN-japanese model)
  • grounding-dino — BertTokenizer or BertTokenizerFast (Grounding DINO model)
  • groupvit — CLIPTokenizer or CLIPTokenizerFast (GroupViT model)
  • herbert — HerbertTokenizer or HerbertTokenizerFast (HerBERT model)
  • hubert — Wav2Vec2CTCTokenizer (Hubert model)
  • ibert — RobertaTokenizer or RobertaTokenizerFast (I-BERT model)
  • idefics — LlamaTokenizerFast (IDEFICS model)
  • idefics2 — LlamaTokenizer or LlamaTokenizerFast (Idefics2 model)
  • instructblip — GPT2Tokenizer or GPT2TokenizerFast (InstructBLIP model)
  • jamba — LlamaTokenizer or LlamaTokenizerFast (Jamba model)
  • jukebox — JukeboxTokenizer (Jukebox model)
  • kosmos-2 — XLMRobertaTokenizer or XLMRobertaTokenizerFast (KOSMOS-2 model)
  • layoutlm — LayoutLMTokenizer or LayoutLMTokenizerFast (LayoutLM model)
  • layoutlmv2 — LayoutLMv2Tokenizer or LayoutLMv2TokenizerFast (LayoutLMv2 model)
  • layoutlmv3 — LayoutLMv3Tokenizer or LayoutLMv3TokenizerFast (LayoutLMv3 model)
  • layoutxlm — LayoutXLMTokenizer or LayoutXLMTokenizerFast (LayoutXLM model)
  • led — LEDTokenizer or LEDTokenizerFast (LED model)
  • lilt — LayoutLMv3Tokenizer or LayoutLMv3TokenizerFast (LiLT model)
  • llama — LlamaTokenizer or LlamaTokenizerFast (LLaMA model)
  • llava — LlamaTokenizer or LlamaTokenizerFast (LLaVa model)
  • llava_next — LlamaTokenizer or LlamaTokenizerFast (LLaVA-NeXT model)
  • longformer — LongformerTokenizer or LongformerTokenizerFast (Longformer model)
  • longt5 — T5Tokenizer or T5TokenizerFast (LongT5 model)
  • luke — LukeTokenizer (LUKE model)
  • lxmert — LxmertTokenizer or LxmertTokenizerFast (LXMERT model)
  • m2m_100 — M2M100Tokenizer (M2M100 model)
  • mamba — GPTNeoXTokenizerFast (Mamba model)
  • marian — MarianTokenizer (Marian model)
  • mbart — MBartTokenizer or MBartTokenizerFast (mBART model)
  • mbart50 — MBart50Tokenizer or MBart50TokenizerFast (mBART-50 model)
  • mega — RobertaTokenizer or RobertaTokenizerFast (MEGA model)
  • megatron-bert — BertTokenizer or BertTokenizerFast (Megatron-BERT model)
  • mgp-str — MgpstrTokenizer (MGP-STR model)
  • mistral — LlamaTokenizer or LlamaTokenizerFast (Mistral model)
  • mixtral — LlamaTokenizer or LlamaTokenizerFast (Mixtral model)
  • mluke — MLukeTokenizer (mLUKE model)
  • mobilebert — MobileBertTokenizer or MobileBertTokenizerFast (MobileBERT model)
  • mpnet — MPNetTokenizer or MPNetTokenizerFast (MPNet model)
  • mpt — GPTNeoXTokenizerFast (MPT model)
  • mra — RobertaTokenizer or RobertaTokenizerFast (MRA model)
  • mt5 — MT5Tokenizer or MT5TokenizerFast (MT5 model)
  • musicgen — T5Tokenizer or T5TokenizerFast (MusicGen model)
  • musicgen_melody — T5Tokenizer or T5TokenizerFast (MusicGen Melody model)
  • mvp — MvpTokenizer or MvpTokenizerFast (MVP model)
  • nezha — BertTokenizer or BertTokenizerFast (Nezha model)
  • nllb — NllbTokenizer or NllbTokenizerFast (NLLB model)
  • nllb-moe — NllbTokenizer or NllbTokenizerFast (NLLB-MOE model)
  • nystromformer — AlbertTokenizer or AlbertTokenizerFast (Nyströmformer model)
  • olmo — GPTNeoXTokenizerFast (OLMo model)
  • oneformer — CLIPTokenizer or CLIPTokenizerFast (OneFormer model)
  • openai-gpt — OpenAIGPTTokenizer or OpenAIGPTTokenizerFast (OpenAI GPT model)
  • opt — GPT2Tokenizer or GPT2TokenizerFast (OPT model)
  • owlv2 — CLIPTokenizer or CLIPTokenizerFast (OWLv2 model)
  • owlvit — CLIPTokenizer or CLIPTokenizerFast (OWL-ViT model)
  • pegasus — PegasusTokenizer or PegasusTokenizerFast (Pegasus model)
  • pegasus_x — PegasusTokenizer or PegasusTokenizerFast (PEGASUS-X model)
  • perceiver — PerceiverTokenizer (Perceiver model)
  • persimmon — LlamaTokenizer or LlamaTokenizerFast (Persimmon model)
  • phi — CodeGenTokenizer or CodeGenTokenizerFast (Phi model)
  • phobert — PhobertTokenizer (PhoBERT model)
  • pix2struct — T5Tokenizer or T5TokenizerFast (Pix2Struct model)
  • plbart — PLBartTokenizer (PLBart model)
  • prophetnet — ProphetNetTokenizer (ProphetNet model)
  • qdqbert — BertTokenizer or BertTokenizerFast (QDQBert model)
  • qwen2 — Qwen2Tokenizer or Qwen2TokenizerFast (Qwen2 model)
  • qwen2_moe — Qwen2Tokenizer or Qwen2TokenizerFast (Qwen2MoE model)
  • rag — RagTokenizer (RAG model)
  • realm — RealmTokenizer or RealmTokenizerFast (REALM model)
  • recurrent_gemma — GemmaTokenizer or GemmaTokenizerFast (RecurrentGemma model)
  • reformer — ReformerTokenizer or ReformerTokenizerFast (Reformer model)
  • rembert — RemBertTokenizer or RemBertTokenizerFast (RemBERT model)
  • retribert — RetriBertTokenizer or RetriBertTokenizerFast (RetriBERT model)
  • roberta — RobertaTokenizer or RobertaTokenizerFast (RoBERTa model)
  • roberta-prelayernorm — RobertaTokenizer or RobertaTokenizerFast (RoBERTa-PreLayerNorm model)
  • roc_bert — RoCBertTokenizer (RoCBert model)
  • roformer — RoFormerTokenizer or RoFormerTokenizerFast (RoFormer model)
  • rwkv — GPTNeoXTokenizerFast (RWKV model)
  • seamless_m4t — SeamlessM4TTokenizer or SeamlessM4TTokenizerFast (SeamlessM4T model)
  • seamless_m4t_v2 — SeamlessM4TTokenizer or SeamlessM4TTokenizerFast (SeamlessM4Tv2 model)
  • siglip — SiglipTokenizer (SigLIP model)
  • speech_to_text — Speech2TextTokenizer (Speech2Text model)
  • speech_to_text_2 — Speech2Text2Tokenizer (Speech2Text2 model)
  • speecht5 — SpeechT5Tokenizer (SpeechT5 model)
  • splinter — SplinterTokenizer or SplinterTokenizerFast (Splinter model)
  • squeezebert — SqueezeBertTokenizer or SqueezeBertTokenizerFast (SqueezeBERT model)
  • stablelm — GPTNeoXTokenizerFast (StableLm model)
  • starcoder2 — GPT2Tokenizer or GPT2TokenizerFast (Starcoder2 model)
  • switch_transformers — T5Tokenizer or T5TokenizerFast (SwitchTransformers model)
  • t5 — T5Tokenizer or T5TokenizerFast (T5 model)
  • tapas — TapasTokenizer (TAPAS model)
  • tapex — TapexTokenizer (TAPEX model)
  • transfo-xl — TransfoXLTokenizer (Transformer-XL model)
  • tvp — BertTokenizer or BertTokenizerFast (TVP model)
  • udop — UdopTokenizer or UdopTokenizerFast (UDOP model)
  • umt5 — T5Tokenizer or T5TokenizerFast (UMT5 model)
  • vilt — BertTokenizer or BertTokenizerFast (ViLT model)
  • vipllava — LlamaTokenizer or LlamaTokenizerFast (VipLlava model)
  • visual_bert — BertTokenizer or BertTokenizerFast (VisualBERT model)
  • vits — VitsTokenizer (VITS model)
  • wav2vec2 — Wav2Vec2CTCTokenizer (Wav2Vec2 model)
  • wav2vec2-bert — Wav2Vec2CTCTokenizer (Wav2Vec2-BERT model)
  • wav2vec2-conformer — Wav2Vec2CTCTokenizer (Wav2Vec2-Conformer model)
  • wav2vec2_phoneme — Wav2Vec2PhonemeCTCTokenizer (Wav2Vec2Phoneme model)
  • whisper — WhisperTokenizer or WhisperTokenizerFast (Whisper model)
  • xclip — CLIPTokenizer or CLIPTokenizerFast (X-CLIP model)
  • xglm — XGLMTokenizer or XGLMTokenizerFast (XGLM model)
  • xlm — XLMTokenizer (XLM model)
  • xlm-prophetnet — XLMProphetNetTokenizer (XLM-ProphetNet model)
  • xlm-roberta — XLMRobertaTokenizer or XLMRobertaTokenizerFast (XLM-RoBERTa model)
  • xlm-roberta-xl — XLMRobertaTokenizer or XLMRobertaTokenizerFast (XLM-RoBERTa-XL model)
  • xlnet — XLNetTokenizer or XLNetTokenizerFast (XLNet model)
  • xmod — XLMRobertaTokenizer or XLMRobertaTokenizerFast (X-MOD model)
  • yoso — AlbertTokenizer or AlbertTokenizerFast (YOSO model)

Examples:

>>> from transformers import AutoTokenizer

>>> # Download vocabulary from huggingface.co and cache.
>>> tokenizer = AutoTokenizer.from_pretrained("google-bert/bert-base-uncased")

>>> # Download vocabulary from huggingface.co (user-uploaded) and cache.
>>> tokenizer = AutoTokenizer.from_pretrained("dbmdz/bert-base-german-cased")

>>> # If vocabulary files are in a directory (e.g. tokenizer was saved using *save_pretrained('./test/saved_model/')*)
>>> # tokenizer = AutoTokenizer.from_pretrained("./test/bert_saved_model/")

>>> # Download vocabulary from huggingface.co and define model-specific arguments
>>> tokenizer = AutoTokenizer.from_pretrained("FacebookAI/roberta-base", add_prefix_space=True)

register

< >

( config_class slow_tokenizer_class = None fast_tokenizer_class = None exist_ok = False )

Parameters

  • config_class (PretrainedConfig) — The configuration corresponding to the model to register.
  • slow_tokenizer_class (PretrainedTokenizer, optional) — The slow tokenizer to register.
  • fast_tokenizer_class (PretrainedTokenizerFast, optional) — The fast tokenizer to register.

Register a new tokenizer in this mapping.

AutoFeatureExtractor

class transformers.AutoFeatureExtractor

< >

( )

This is a generic feature extractor class that will be instantiated as one of the feature extractor classes of the library when created with the AutoFeatureExtractor.from_pretrained() class method.

This class cannot be instantiated directly using __init__() (throws an error).

from_pretrained

< >

( pretrained_model_name_or_path **kwargs )

Parameters

  • pretrained_model_name_or_path (str or os.PathLike) — This can be either:

    • a string, the model id of a pretrained feature_extractor hosted inside a model repo on huggingface.co.
    • a path to a directory containing a feature extractor file saved using the save_pretrained() method, e.g., ./my_model_directory/.
    • a path or url to a saved feature extractor JSON file, e.g., ./my_model_directory/preprocessor_config.json.
  • cache_dir (str or os.PathLike, optional) — Path to a directory in which a downloaded pretrained model feature extractor should be cached if the standard cache should not be used.
  • force_download (bool, optional, defaults to False) — Whether or not to force to (re-)download the feature extractor files and override the cached versions if they exist.
  • resume_download (bool, optional, defaults to False) — Whether or not to delete incompletely received file. Attempts to resume the download if such a file exists.
  • proxies (Dict[str, str], optional) — A dictionary of proxy servers to use by protocol or endpoint, e.g., {'http': 'foo.bar:3128', 'http://hostname': 'foo.bar:4012'}. The proxies are used on each request.
  • token (str or bool, optional) — The token to use as HTTP bearer authorization for remote files. If True, will use the token generated when running huggingface-cli login (stored in ~/.huggingface).
  • revision (str, optional, defaults to "main") — The specific model version to use. It can be a branch name, a tag name, or a commit id, since we use a git-based system for storing models and other artifacts on huggingface.co, so revision can be any identifier allowed by git.
  • return_unused_kwargs (bool, optional, defaults to False) — If False, then this function returns just the final feature extractor object. If True, then this functions returns a Tuple(feature_extractor, unused_kwargs) where unused_kwargs is a dictionary consisting of the key/value pairs whose keys are not feature extractor attributes: i.e., the part of kwargs which has not been used to update feature_extractor and is otherwise ignored.
  • trust_remote_code (bool, optional, defaults to False) — Whether or not to allow for custom models defined on the Hub in their own modeling files. This option should only be set to True for repositories you trust and in which you have read the code, as it will execute code present on the Hub on your local machine.
  • kwargs (Dict[str, Any], optional) — The values in kwargs of any keys which are feature extractor attributes will be used to override the loaded values. Behavior concerning key/value pairs whose keys are not feature extractor attributes is controlled by the return_unused_kwargs keyword parameter.

Instantiate one of the feature extractor classes of the library from a pretrained model vocabulary.

The feature extractor class to instantiate is selected based on the model_type property of the config object (either passed as an argument or loaded from pretrained_model_name_or_path if possible), or when it’s missing, by falling back to using pattern matching on pretrained_model_name_or_path:

  • audio-spectrogram-transformer — ASTFeatureExtractor (Audio Spectrogram Transformer model)
  • beit — BeitFeatureExtractor (BEiT model)
  • chinese_clip — ChineseCLIPFeatureExtractor (Chinese-CLIP model)
  • clap — ClapFeatureExtractor (CLAP model)
  • clip — CLIPFeatureExtractor (CLIP model)
  • clipseg — ViTFeatureExtractor (CLIPSeg model)
  • clvp — ClvpFeatureExtractor (CLVP model)
  • conditional_detr — ConditionalDetrFeatureExtractor (Conditional DETR model)
  • convnext — ConvNextFeatureExtractor (ConvNeXT model)
  • cvt — ConvNextFeatureExtractor (CvT model)
  • data2vec-audio — Wav2Vec2FeatureExtractor (Data2VecAudio model)
  • data2vec-vision — BeitFeatureExtractor (Data2VecVision model)
  • deformable_detr — DeformableDetrFeatureExtractor (Deformable DETR model)
  • deit — DeiTFeatureExtractor (DeiT model)
  • detr — DetrFeatureExtractor (DETR model)
  • dinat — ViTFeatureExtractor (DiNAT model)
  • donut-swin — DonutFeatureExtractor (DonutSwin model)
  • dpt — DPTFeatureExtractor (DPT model)
  • encodec — EncodecFeatureExtractor (EnCodec model)
  • flava — FlavaFeatureExtractor (FLAVA model)
  • glpn — GLPNFeatureExtractor (GLPN model)
  • groupvit — CLIPFeatureExtractor (GroupViT model)
  • hubert — Wav2Vec2FeatureExtractor (Hubert model)
  • imagegpt — ImageGPTFeatureExtractor (ImageGPT model)
  • layoutlmv2 — LayoutLMv2FeatureExtractor (LayoutLMv2 model)
  • layoutlmv3 — LayoutLMv3FeatureExtractor (LayoutLMv3 model)
  • levit — LevitFeatureExtractor (LeViT model)
  • maskformer — MaskFormerFeatureExtractor (MaskFormer model)
  • mctct — MCTCTFeatureExtractor (M-CTC-T model)
  • mobilenet_v1 — MobileNetV1FeatureExtractor (MobileNetV1 model)
  • mobilenet_v2 — MobileNetV2FeatureExtractor (MobileNetV2 model)
  • mobilevit — MobileViTFeatureExtractor (MobileViT model)
  • nat — ViTFeatureExtractor (NAT model)
  • owlvit — OwlViTFeatureExtractor (OWL-ViT model)
  • perceiver — PerceiverFeatureExtractor (Perceiver model)
  • poolformer — PoolFormerFeatureExtractor (PoolFormer model)
  • pop2piano — Pop2PianoFeatureExtractor (Pop2Piano model)
  • regnet — ConvNextFeatureExtractor (RegNet model)
  • resnet — ConvNextFeatureExtractor (ResNet model)
  • seamless_m4t — SeamlessM4TFeatureExtractor (SeamlessM4T model)
  • seamless_m4t_v2 — SeamlessM4TFeatureExtractor (SeamlessM4Tv2 model)
  • segformer — SegformerFeatureExtractor (SegFormer model)
  • sew — Wav2Vec2FeatureExtractor (SEW model)
  • sew-d — Wav2Vec2FeatureExtractor (SEW-D model)
  • speech_to_text — Speech2TextFeatureExtractor (Speech2Text model)
  • speecht5 — SpeechT5FeatureExtractor (SpeechT5 model)
  • swiftformer — ViTFeatureExtractor (SwiftFormer model)
  • swin — ViTFeatureExtractor (Swin Transformer model)
  • swinv2 — ViTFeatureExtractor (Swin Transformer V2 model)
  • table-transformer — DetrFeatureExtractor (Table Transformer model)
  • timesformer — VideoMAEFeatureExtractor (TimeSformer model)
  • tvlt — TvltFeatureExtractor (TVLT model)
  • unispeech — Wav2Vec2FeatureExtractor (UniSpeech model)
  • unispeech-sat — Wav2Vec2FeatureExtractor (UniSpeechSat model)
  • univnet — UnivNetFeatureExtractor (UnivNet model)
  • van — ConvNextFeatureExtractor (VAN model)
  • videomae — VideoMAEFeatureExtractor (VideoMAE model)
  • vilt — ViltFeatureExtractor (ViLT model)
  • vit — ViTFeatureExtractor (ViT model)
  • vit_mae — ViTFeatureExtractor (ViTMAE model)
  • vit_msn — ViTFeatureExtractor (ViTMSN model)
  • wav2vec2 — Wav2Vec2FeatureExtractor (Wav2Vec2 model)
  • wav2vec2-bert — Wav2Vec2FeatureExtractor (Wav2Vec2-BERT model)
  • wav2vec2-conformer — Wav2Vec2FeatureExtractor (Wav2Vec2-Conformer model)
  • wavlm — Wav2Vec2FeatureExtractor (WavLM model)
  • whisper — WhisperFeatureExtractor (Whisper model)
  • xclip — CLIPFeatureExtractor (X-CLIP model)
  • yolos — YolosFeatureExtractor (YOLOS model)

Passing token=True is required when you want to use a private model.

Examples:

>>> from transformers import AutoFeatureExtractor

>>> # Download feature extractor from huggingface.co and cache.
>>> feature_extractor = AutoFeatureExtractor.from_pretrained("facebook/wav2vec2-base-960h")

>>> # If feature extractor files are in a directory (e.g. feature extractor was saved using *save_pretrained('./test/saved_model/')*)
>>> # feature_extractor = AutoFeatureExtractor.from_pretrained("./test/saved_model/")

register

< >

( config_class feature_extractor_class exist_ok = False )

Parameters

  • config_class (PretrainedConfig) — The configuration corresponding to the model to register.
  • feature_extractor_class (FeatureExtractorMixin) — The feature extractor to register.

Register a new feature extractor for this class.

AutoImageProcessor

class transformers.AutoImageProcessor

< >

( )

This is a generic image processor class that will be instantiated as one of the image processor classes of the library when created with the AutoImageProcessor.from_pretrained() class method.

This class cannot be instantiated directly using __init__() (throws an error).

from_pretrained

< >

( pretrained_model_name_or_path **kwargs )

Parameters

  • pretrained_model_name_or_path (str or os.PathLike) — This can be either:

    • a string, the model id of a pretrained image_processor hosted inside a model repo on huggingface.co.
    • a path to a directory containing a image processor file saved using the save_pretrained() method, e.g., ./my_model_directory/.
    • a path or url to a saved image processor JSON file, e.g., ./my_model_directory/preprocessor_config.json.
  • cache_dir (str or os.PathLike, optional) — Path to a directory in which a downloaded pretrained model image processor should be cached if the standard cache should not be used.
  • force_download (bool, optional, defaults to False) — Whether or not to force to (re-)download the image processor files and override the cached versions if they exist.
  • resume_download (bool, optional, defaults to False) — Whether or not to delete incompletely received file. Attempts to resume the download if such a file exists.
  • proxies (Dict[str, str], optional) — A dictionary of proxy servers to use by protocol or endpoint, e.g., {'http': 'foo.bar:3128', 'http://hostname': 'foo.bar:4012'}. The proxies are used on each request.
  • token (str or bool, optional) — The token to use as HTTP bearer authorization for remote files. If True, will use the token generated when running huggingface-cli login (stored in ~/.huggingface).
  • revision (str, optional, defaults to "main") — The specific model version to use. It can be a branch name, a tag name, or a commit id, since we use a git-based system for storing models and other artifacts on huggingface.co, so revision can be any identifier allowed by git.
  • return_unused_kwargs (bool, optional, defaults to False) — If False, then this function returns just the final image processor object. If True, then this functions returns a Tuple(image_processor, unused_kwargs) where unused_kwargs is a dictionary consisting of the key/value pairs whose keys are not image processor attributes: i.e., the part of kwargs which has not been used to update image_processor and is otherwise ignored.
  • trust_remote_code (bool, optional, defaults to False) — Whether or not to allow for custom models defined on the Hub in their own modeling files. This option should only be set to True for repositories you trust and in which you have read the code, as it will execute code present on the Hub on your local machine.
  • kwargs (Dict[str, Any], optional) — The values in kwargs of any keys which are image processor attributes will be used to override the loaded values. Behavior concerning key/value pairs whose keys are not image processor attributes is controlled by the return_unused_kwargs keyword parameter.

Instantiate one of the image processor classes of the library from a pretrained model vocabulary.

The image processor class to instantiate is selected based on the model_type property of the config object (either passed as an argument or loaded from pretrained_model_name_or_path if possible), or when it’s missing, by falling back to using pattern matching on pretrained_model_name_or_path:

  • align — EfficientNetImageProcessor (ALIGN model)
  • beit — BeitImageProcessor (BEiT model)
  • bit — BitImageProcessor (BiT model)
  • blip — BlipImageProcessor (BLIP model)
  • blip-2 — BlipImageProcessor (BLIP-2 model)
  • bridgetower — BridgeTowerImageProcessor (BridgeTower model)
  • chinese_clip — ChineseCLIPImageProcessor (Chinese-CLIP model)
  • clip — CLIPImageProcessor (CLIP model)
  • clipseg — ViTImageProcessor (CLIPSeg model)
  • conditional_detr — ConditionalDetrImageProcessor (Conditional DETR model)
  • convnext — ConvNextImageProcessor (ConvNeXT model)
  • convnextv2 — ConvNextImageProcessor (ConvNeXTV2 model)
  • cvt — ConvNextImageProcessor (CvT model)
  • data2vec-vision — BeitImageProcessor (Data2VecVision model)
  • deformable_detr — DeformableDetrImageProcessor (Deformable DETR model)
  • deit — DeiTImageProcessor (DeiT model)
  • depth_anything — DPTImageProcessor (Depth Anything model)
  • deta — DetaImageProcessor (DETA model)
  • detr — DetrImageProcessor (DETR model)
  • dinat — ViTImageProcessor (DiNAT model)
  • dinov2 — BitImageProcessor (DINOv2 model)
  • donut-swin — DonutImageProcessor (DonutSwin model)
  • dpt — DPTImageProcessor (DPT model)
  • efficientformer — EfficientFormerImageProcessor (EfficientFormer model)
  • efficientnet — EfficientNetImageProcessor (EfficientNet model)
  • flava — FlavaImageProcessor (FLAVA model)
  • focalnet — BitImageProcessor (FocalNet model)
  • fuyu — FuyuImageProcessor (Fuyu model)
  • git — CLIPImageProcessor (GIT model)
  • glpn — GLPNImageProcessor (GLPN model)
  • grounding-dino — GroundingDinoImageProcessor (Grounding DINO model)
  • groupvit — CLIPImageProcessor (GroupViT model)
  • idefics — IdeficsImageProcessor (IDEFICS model)
  • idefics2 — Idefics2ImageProcessor (Idefics2 model)
  • imagegpt — ImageGPTImageProcessor (ImageGPT model)
  • instructblip — BlipImageProcessor (InstructBLIP model)
  • kosmos-2 — CLIPImageProcessor (KOSMOS-2 model)
  • layoutlmv2 — LayoutLMv2ImageProcessor (LayoutLMv2 model)
  • layoutlmv3 — LayoutLMv3ImageProcessor (LayoutLMv3 model)
  • levit — LevitImageProcessor (LeViT model)
  • llava — CLIPImageProcessor (LLaVa model)
  • llava_next — LlavaNextImageProcessor (LLaVA-NeXT model)
  • mask2former — Mask2FormerImageProcessor (Mask2Former model)
  • maskformer — MaskFormerImageProcessor (MaskFormer model)
  • mgp-str — ViTImageProcessor (MGP-STR model)
  • mobilenet_v1 — MobileNetV1ImageProcessor (MobileNetV1 model)
  • mobilenet_v2 — MobileNetV2ImageProcessor (MobileNetV2 model)
  • mobilevit — MobileViTImageProcessor (MobileViT model)
  • mobilevitv2 — MobileViTImageProcessor (MobileViTV2 model)
  • nat — ViTImageProcessor (NAT model)
  • nougat — NougatImageProcessor (Nougat model)
  • oneformer — OneFormerImageProcessor (OneFormer model)
  • owlv2 — Owlv2ImageProcessor (OWLv2 model)
  • owlvit — OwlViTImageProcessor (OWL-ViT model)
  • perceiver — PerceiverImageProcessor (Perceiver model)
  • pix2struct — Pix2StructImageProcessor (Pix2Struct model)
  • poolformer — PoolFormerImageProcessor (PoolFormer model)
  • pvt — PvtImageProcessor (PVT model)
  • pvt_v2 — PvtImageProcessor (PVTv2 model)
  • regnet — ConvNextImageProcessor (RegNet model)
  • resnet — ConvNextImageProcessor (ResNet model)
  • sam — SamImageProcessor (SAM model)
  • segformer — SegformerImageProcessor (SegFormer model)
  • seggpt — SegGptImageProcessor (SegGPT model)
  • siglip — SiglipImageProcessor (SigLIP model)
  • swiftformer — ViTImageProcessor (SwiftFormer model)
  • swin — ViTImageProcessor (Swin Transformer model)
  • swin2sr — Swin2SRImageProcessor (Swin2SR model)
  • swinv2 — ViTImageProcessor (Swin Transformer V2 model)
  • table-transformer — DetrImageProcessor (Table Transformer model)
  • timesformer — VideoMAEImageProcessor (TimeSformer model)
  • tvlt — TvltImageProcessor (TVLT model)
  • tvp — TvpImageProcessor (TVP model)
  • udop — LayoutLMv3ImageProcessor (UDOP model)
  • upernet — SegformerImageProcessor (UPerNet model)
  • van — ConvNextImageProcessor (VAN model)
  • videomae — VideoMAEImageProcessor (VideoMAE model)
  • vilt — ViltImageProcessor (ViLT model)
  • vipllava — CLIPImageProcessor (VipLlava model)
  • vit — ViTImageProcessor (ViT model)
  • vit_hybrid — ViTHybridImageProcessor (ViT Hybrid model)
  • vit_mae — ViTImageProcessor (ViTMAE model)
  • vit_msn — ViTImageProcessor (ViTMSN model)
  • vitmatte — VitMatteImageProcessor (ViTMatte model)
  • xclip — CLIPImageProcessor (X-CLIP model)
  • yolos — YolosImageProcessor (YOLOS model)

Passing token=True is required when you want to use a private model.

Examples:

>>> from transformers import AutoImageProcessor

>>> # Download image processor from huggingface.co and cache.
>>> image_processor = AutoImageProcessor.from_pretrained("google/vit-base-patch16-224-in21k")

>>> # If image processor files are in a directory (e.g. image processor was saved using *save_pretrained('./test/saved_model/')*)
>>> # image_processor = AutoImageProcessor.from_pretrained("./test/saved_model/")

register

< >

( config_class image_processor_class exist_ok = False )

Parameters

Register a new image processor for this class.

AutoProcessor

class transformers.AutoProcessor

< >

( )

This is a generic processor class that will be instantiated as one of the processor classes of the library when created with the AutoProcessor.from_pretrained() class method.

This class cannot be instantiated directly using __init__() (throws an error).

from_pretrained

< >

( pretrained_model_name_or_path **kwargs )

Parameters

  • pretrained_model_name_or_path (str or os.PathLike) — This can be either:

    • a string, the model id of a pretrained feature_extractor hosted inside a model repo on huggingface.co.
    • a path to a directory containing a processor files saved using the save_pretrained() method, e.g., ./my_model_directory/.
  • cache_dir (str or os.PathLike, optional) — Path to a directory in which a downloaded pretrained model feature extractor should be cached if the standard cache should not be used.
  • force_download (bool, optional, defaults to False) — Whether or not to force to (re-)download the feature extractor files and override the cached versions if they exist.
  • resume_download (bool, optional, defaults to False) — Whether or not to delete incompletely received file. Attempts to resume the download if such a file exists.
  • proxies (Dict[str, str], optional) — A dictionary of proxy servers to use by protocol or endpoint, e.g., {'http': 'foo.bar:3128', 'http://hostname': 'foo.bar:4012'}. The proxies are used on each request.
  • token (str or bool, optional) — The token to use as HTTP bearer authorization for remote files. If True, will use the token generated when running huggingface-cli login (stored in ~/.huggingface).
  • revision (str, optional, defaults to "main") — The specific model version to use. It can be a branch name, a tag name, or a commit id, since we use a git-based system for storing models and other artifacts on huggingface.co, so revision can be any identifier allowed by git.
  • return_unused_kwargs (bool, optional, defaults to False) — If False, then this function returns just the final feature extractor object. If True, then this functions returns a Tuple(feature_extractor, unused_kwargs) where unused_kwargs is a dictionary consisting of the key/value pairs whose keys are not feature extractor attributes: i.e., the part of kwargs which has not been used to update feature_extractor and is otherwise ignored.
  • trust_remote_code (bool, optional, defaults to False) — Whether or not to allow for custom models defined on the Hub in their own modeling files. This option should only be set to True for repositories you trust and in which you have read the code, as it will execute code present on the Hub on your local machine.
  • kwargs (Dict[str, Any], optional) — The values in kwargs of any keys which are feature extractor attributes will be used to override the loaded values. Behavior concerning key/value pairs whose keys are not feature extractor attributes is controlled by the return_unused_kwargs keyword parameter.

Instantiate one of the processor classes of the library from a pretrained model vocabulary.

The processor class to instantiate is selected based on the model_type property of the config object (either passed as an argument or loaded from pretrained_model_name_or_path if possible):

  • align — AlignProcessor (ALIGN model)
  • altclip — AltCLIPProcessor (AltCLIP model)
  • bark — BarkProcessor (Bark model)
  • blip — BlipProcessor (BLIP model)
  • blip-2 — Blip2Processor (BLIP-2 model)
  • bridgetower — BridgeTowerProcessor (BridgeTower model)
  • chinese_clip — ChineseCLIPProcessor (Chinese-CLIP model)
  • clap — ClapProcessor (CLAP model)
  • clip — CLIPProcessor (CLIP model)
  • clipseg — CLIPSegProcessor (CLIPSeg model)
  • clvp — ClvpProcessor (CLVP model)
  • flava — FlavaProcessor (FLAVA model)
  • fuyu — FuyuProcessor (Fuyu model)
  • git — GitProcessor (GIT model)
  • groupvit — CLIPProcessor (GroupViT model)
  • hubert — Wav2Vec2Processor (Hubert model)
  • idefics — IdeficsProcessor (IDEFICS model)
  • idefics2 — Idefics2Processor (Idefics2 model)
  • instructblip — InstructBlipProcessor (InstructBLIP model)
  • kosmos-2 — Kosmos2Processor (KOSMOS-2 model)
  • layoutlmv2 — LayoutLMv2Processor (LayoutLMv2 model)
  • layoutlmv3 — LayoutLMv3Processor (LayoutLMv3 model)
  • llava — LlavaProcessor (LLaVa model)
  • llava_next — LlavaNextProcessor (LLaVA-NeXT model)
  • markuplm — MarkupLMProcessor (MarkupLM model)
  • mctct — MCTCTProcessor (M-CTC-T model)
  • mgp-str — MgpstrProcessor (MGP-STR model)
  • oneformer — OneFormerProcessor (OneFormer model)
  • owlv2 — Owlv2Processor (OWLv2 model)
  • owlvit — OwlViTProcessor (OWL-ViT model)
  • pix2struct — Pix2StructProcessor (Pix2Struct model)
  • pop2piano — Pop2PianoProcessor (Pop2Piano model)
  • sam — SamProcessor (SAM model)
  • seamless_m4t — SeamlessM4TProcessor (SeamlessM4T model)
  • sew — Wav2Vec2Processor (SEW model)
  • sew-d — Wav2Vec2Processor (SEW-D model)
  • siglip — SiglipProcessor (SigLIP model)
  • speech_to_text — Speech2TextProcessor (Speech2Text model)
  • speech_to_text_2 — Speech2Text2Processor (Speech2Text2 model)
  • speecht5 — SpeechT5Processor (SpeechT5 model)
  • trocr — TrOCRProcessor (TrOCR model)
  • tvlt — TvltProcessor (TVLT model)
  • tvp — TvpProcessor (TVP model)
  • unispeech — Wav2Vec2Processor (UniSpeech model)
  • unispeech-sat — Wav2Vec2Processor (UniSpeechSat model)
  • vilt — ViltProcessor (ViLT model)
  • vipllava — LlavaProcessor (VipLlava model)
  • vision-text-dual-encoder — VisionTextDualEncoderProcessor (VisionTextDualEncoder model)
  • wav2vec2 — Wav2Vec2Processor (Wav2Vec2 model)
  • wav2vec2-bert — Wav2Vec2Processor (Wav2Vec2-BERT model)
  • wav2vec2-conformer — Wav2Vec2Processor (Wav2Vec2-Conformer model)
  • wavlm — Wav2Vec2Processor (WavLM model)
  • whisper — WhisperProcessor (Whisper model)
  • xclip — XCLIPProcessor (X-CLIP model)

Passing token=True is required when you want to use a private model.

Examples:

>>> from transformers import AutoProcessor

>>> # Download processor from huggingface.co and cache.
>>> processor = AutoProcessor.from_pretrained("facebook/wav2vec2-base-960h")

>>> # If processor files are in a directory (e.g. processor was saved using *save_pretrained('./test/saved_model/')*)
>>> # processor = AutoProcessor.from_pretrained("./test/saved_model/")

register

< >

( config_class processor_class exist_ok = False )

Parameters

  • config_class (PretrainedConfig) — The configuration corresponding to the model to register.
  • processor_class (FeatureExtractorMixin) — The processor to register.

Register a new processor for this class.

Generic model classes

以下の自動クラスは、特定のヘッドを持たないベースモデルクラスをインスタンス化するために利用可能です。

AutoModel

class transformers.AutoModel

< >

( *args **kwargs )

This is a generic model class that will be instantiated as one of the base model classes of the library when created with the from_pretrained() class method or the from_config() class method.

This class cannot be instantiated directly using __init__() (throws an error).

from_config

< >

( **kwargs )

Parameters

  • config (PretrainedConfig) — The model class to instantiate is selected based on the configuration class:

    • ASTConfig configuration class: ASTModel (Audio Spectrogram Transformer model)
    • AlbertConfig configuration class: AlbertModel (ALBERT model)
    • AlignConfig configuration class: AlignModel (ALIGN model)
    • AltCLIPConfig configuration class: AltCLIPModel (AltCLIP model)
    • AutoformerConfig configuration class: AutoformerModel (Autoformer model)
    • BarkConfig configuration class: BarkModel (Bark model)
    • BartConfig configuration class: BartModel (BART model)
    • BeitConfig configuration class: BeitModel (BEiT model)
    • BertConfig configuration class: BertModel (BERT model)
    • BertGenerationConfig configuration class: BertGenerationEncoder (Bert Generation model)
    • BigBirdConfig configuration class: BigBirdModel (BigBird model)
    • BigBirdPegasusConfig configuration class: BigBirdPegasusModel (BigBird-Pegasus model)
    • BioGptConfig configuration class: BioGptModel (BioGpt model)
    • BitConfig configuration class: BitModel (BiT model)
    • BlenderbotConfig configuration class: BlenderbotModel (Blenderbot model)
    • BlenderbotSmallConfig configuration class: BlenderbotSmallModel (BlenderbotSmall model)
    • Blip2Config configuration class: Blip2Model (BLIP-2 model)
    • BlipConfig configuration class: BlipModel (BLIP model)
    • BloomConfig configuration class: BloomModel (BLOOM model)
    • BridgeTowerConfig configuration class: BridgeTowerModel (BridgeTower model)
    • BrosConfig configuration class: BrosModel (BROS model)
    • CLIPConfig configuration class: CLIPModel (CLIP model)
    • CLIPSegConfig configuration class: CLIPSegModel (CLIPSeg model)
    • CLIPVisionConfig configuration class: CLIPVisionModel (CLIPVisionModel model)
    • CTRLConfig configuration class: CTRLModel (CTRL model)
    • CamembertConfig configuration class: CamembertModel (CamemBERT model)
    • CanineConfig configuration class: CanineModel (CANINE model)
    • ChineseCLIPConfig configuration class: ChineseCLIPModel (Chinese-CLIP model)
    • ChineseCLIPVisionConfig configuration class: ChineseCLIPVisionModel (ChineseCLIPVisionModel model)
    • ClapConfig configuration class: ClapModel (CLAP model)
    • ClvpConfig configuration class: ClvpModelForConditionalGeneration (CLVP model)
    • CodeGenConfig configuration class: CodeGenModel (CodeGen model)
    • CohereConfig configuration class: CohereModel (Cohere model)
    • ConditionalDetrConfig configuration class: ConditionalDetrModel (Conditional DETR model)
    • ConvBertConfig configuration class: ConvBertModel (ConvBERT model)
    • ConvNextConfig configuration class: ConvNextModel (ConvNeXT model)
    • ConvNextV2Config configuration class: ConvNextV2Model (ConvNeXTV2 model)
    • CpmAntConfig configuration class: CpmAntModel (CPM-Ant model)
    • CvtConfig configuration class: CvtModel (CvT model)
    • DPRConfig configuration class: DPRQuestionEncoder (DPR model)
    • DPTConfig configuration class: DPTModel (DPT model)
    • Data2VecAudioConfig configuration class: Data2VecAudioModel (Data2VecAudio model)
    • Data2VecTextConfig configuration class: Data2VecTextModel (Data2VecText model)
    • Data2VecVisionConfig configuration class: Data2VecVisionModel (Data2VecVision model)
    • DbrxConfig configuration class: DbrxModel (DBRX model)
    • DebertaConfig configuration class: DebertaModel (DeBERTa model)
    • DebertaV2Config configuration class: DebertaV2Model (DeBERTa-v2 model)
    • DecisionTransformerConfig configuration class: DecisionTransformerModel (Decision Transformer model)
    • DeformableDetrConfig configuration class: DeformableDetrModel (Deformable DETR model)
    • DeiTConfig configuration class: DeiTModel (DeiT model)
    • DetaConfig configuration class: DetaModel (DETA model)
    • DetrConfig configuration class: DetrModel (DETR model)
    • DinatConfig configuration class: DinatModel (DiNAT model)
    • Dinov2Config configuration class: Dinov2Model (DINOv2 model)
    • DistilBertConfig configuration class: DistilBertModel (DistilBERT model)
    • DonutSwinConfig configuration class: DonutSwinModel (DonutSwin model)
    • EfficientFormerConfig configuration class: EfficientFormerModel (EfficientFormer model)
    • EfficientNetConfig configuration class: EfficientNetModel (EfficientNet model)
    • ElectraConfig configuration class: ElectraModel (ELECTRA model)
    • EncodecConfig configuration class: EncodecModel (EnCodec model)
    • ErnieConfig configuration class: ErnieModel (ERNIE model)
    • ErnieMConfig configuration class: ErnieMModel (ErnieM model)
    • EsmConfig configuration class: EsmModel (ESM model)
    • FNetConfig configuration class: FNetModel (FNet model)
    • FSMTConfig configuration class: FSMTModel (FairSeq Machine-Translation model)
    • FalconConfig configuration class: FalconModel (Falcon model)
    • FastSpeech2ConformerConfig configuration class: FastSpeech2ConformerModel (FastSpeech2Conformer model)
    • FlaubertConfig configuration class: FlaubertModel (FlauBERT model)
    • FlavaConfig configuration class: FlavaModel (FLAVA model)
    • FocalNetConfig configuration class: FocalNetModel (FocalNet model)
    • FunnelConfig configuration class: FunnelModel or FunnelBaseModel (Funnel Transformer model)
    • GLPNConfig configuration class: GLPNModel (GLPN model)
    • GPT2Config configuration class: GPT2Model (OpenAI GPT-2 model)
    • GPTBigCodeConfig configuration class: GPTBigCodeModel (GPTBigCode model)
    • GPTJConfig configuration class: GPTJModel (GPT-J model)
    • GPTNeoConfig configuration class: GPTNeoModel (GPT Neo model)
    • GPTNeoXConfig configuration class: GPTNeoXModel (GPT NeoX model)
    • GPTNeoXJapaneseConfig configuration class: GPTNeoXJapaneseModel (GPT NeoX Japanese model)
    • GPTSanJapaneseConfig configuration class: GPTSanJapaneseForConditionalGeneration (GPTSAN-japanese model)
    • GemmaConfig configuration class: GemmaModel (Gemma model)
    • GitConfig configuration class: GitModel (GIT model)
    • GraphormerConfig configuration class: GraphormerModel (Graphormer model)
    • GroundingDinoConfig configuration class: GroundingDinoModel (Grounding DINO model)
    • GroupViTConfig configuration class: GroupViTModel (GroupViT model)
    • HubertConfig configuration class: HubertModel (Hubert model)
    • IBertConfig configuration class: IBertModel (I-BERT model)
    • Idefics2Config configuration class: Idefics2Model (Idefics2 model)
    • IdeficsConfig configuration class: IdeficsModel (IDEFICS model)
    • ImageGPTConfig configuration class: ImageGPTModel (ImageGPT model)
    • InformerConfig configuration class: InformerModel (Informer model)
    • JambaConfig configuration class: JambaModel (Jamba model)
    • JukeboxConfig configuration class: JukeboxModel (Jukebox model)
    • Kosmos2Config configuration class: Kosmos2Model (KOSMOS-2 model)
    • LEDConfig configuration class: LEDModel (LED model)
    • LayoutLMConfig configuration class: LayoutLMModel (LayoutLM model)
    • LayoutLMv2Config configuration class: LayoutLMv2Model (LayoutLMv2 model)
    • LayoutLMv3Config configuration class: LayoutLMv3Model (LayoutLMv3 model)
    • LevitConfig configuration class: LevitModel (LeViT model)
    • LiltConfig configuration class: LiltModel (LiLT model)
    • LlamaConfig configuration class: LlamaModel (LLaMA model)
    • LongT5Config configuration class: LongT5Model (LongT5 model)
    • LongformerConfig configuration class: LongformerModel (Longformer model)
    • LukeConfig configuration class: LukeModel (LUKE model)
    • LxmertConfig configuration class: LxmertModel (LXMERT model)
    • M2M100Config configuration class: M2M100Model (M2M100 model)
    • MBartConfig configuration class: MBartModel (mBART model)
    • MCTCTConfig configuration class: MCTCTModel (M-CTC-T model)
    • MPNetConfig configuration class: MPNetModel (MPNet model)
    • MT5Config configuration class: MT5Model (MT5 model)
    • MambaConfig configuration class: MambaModel (Mamba model)
    • MarianConfig configuration class: MarianModel (Marian model)
    • MarkupLMConfig configuration class: MarkupLMModel (MarkupLM model)
    • Mask2FormerConfig configuration class: Mask2FormerModel (Mask2Former model)
    • MaskFormerConfig configuration class: MaskFormerModel (MaskFormer model)
    • MaskFormerSwinConfig configuration class: MaskFormerSwinModel (MaskFormerSwin model)
    • MegaConfig configuration class: MegaModel (MEGA model)
    • MegatronBertConfig configuration class: MegatronBertModel (Megatron-BERT model)
    • MgpstrConfig configuration class: MgpstrForSceneTextRecognition (MGP-STR model)
    • MistralConfig configuration class: MistralModel (Mistral model)
    • MixtralConfig configuration class: MixtralModel (Mixtral model)
    • MobileBertConfig configuration class: MobileBertModel (MobileBERT model)
    • MobileNetV1Config configuration class: MobileNetV1Model (MobileNetV1 model)
    • MobileNetV2Config configuration class: MobileNetV2Model (MobileNetV2 model)
    • MobileViTConfig configuration class: MobileViTModel (MobileViT model)
    • MobileViTV2Config configuration class: MobileViTV2Model (MobileViTV2 model)
    • MptConfig configuration class: MptModel (MPT model)
    • MraConfig configuration class: MraModel (MRA model)
    • MvpConfig configuration class: MvpModel (MVP model)
    • NatConfig configuration class: NatModel (NAT model)
    • NezhaConfig configuration class: NezhaModel (Nezha model)
    • NllbMoeConfig configuration class: NllbMoeModel (NLLB-MOE model)
    • NystromformerConfig configuration class: NystromformerModel (Nyströmformer model)
    • OPTConfig configuration class: OPTModel (OPT model)
    • OlmoConfig configuration class: OlmoModel (OLMo model)
    • OneFormerConfig configuration class: OneFormerModel (OneFormer model)
    • OpenAIGPTConfig configuration class: OpenAIGPTModel (OpenAI GPT model)
    • OpenLlamaConfig configuration class: OpenLlamaModel (OpenLlama model)
    • OwlViTConfig configuration class: OwlViTModel (OWL-ViT model)
    • Owlv2Config configuration class: Owlv2Model (OWLv2 model)
    • PLBartConfig configuration class: PLBartModel (PLBart model)
    • PatchTSMixerConfig configuration class: PatchTSMixerModel (PatchTSMixer model)
    • PatchTSTConfig configuration class: PatchTSTModel (PatchTST model)
    • PegasusConfig configuration class: PegasusModel (Pegasus model)
    • PegasusXConfig configuration class: PegasusXModel (PEGASUS-X model)
    • PerceiverConfig configuration class: PerceiverModel (Perceiver model)
    • PersimmonConfig configuration class: PersimmonModel (Persimmon model)
    • PhiConfig configuration class: PhiModel (Phi model)
    • PoolFormerConfig configuration class: PoolFormerModel (PoolFormer model)
    • ProphetNetConfig configuration class: ProphetNetModel (ProphetNet model)
    • PvtConfig configuration class: PvtModel (PVT model)
    • PvtV2Config configuration class: PvtV2Model (PVTv2 model)
    • QDQBertConfig configuration class: QDQBertModel (QDQBert model)
    • Qwen2Config configuration class: Qwen2Model (Qwen2 model)
    • Qwen2MoeConfig configuration class: Qwen2MoeModel (Qwen2MoE model)
    • RecurrentGemmaConfig configuration class: RecurrentGemmaModel (RecurrentGemma model)
    • ReformerConfig configuration class: ReformerModel (Reformer model)
    • RegNetConfig configuration class: RegNetModel (RegNet model)
    • RemBertConfig configuration class: RemBertModel (RemBERT model)
    • ResNetConfig configuration class: ResNetModel (ResNet model)
    • RetriBertConfig configuration class: RetriBertModel (RetriBERT model)
    • RoCBertConfig configuration class: RoCBertModel (RoCBert model)
    • RoFormerConfig configuration class: RoFormerModel (RoFormer model)
    • RobertaConfig configuration class: RobertaModel (RoBERTa model)
    • RobertaPreLayerNormConfig configuration class: RobertaPreLayerNormModel (RoBERTa-PreLayerNorm model)
    • RwkvConfig configuration class: RwkvModel (RWKV model)
    • SEWConfig configuration class: SEWModel (SEW model)
    • SEWDConfig configuration class: SEWDModel (SEW-D model)
    • SamConfig configuration class: SamModel (SAM model)
    • SeamlessM4TConfig configuration class: SeamlessM4TModel (SeamlessM4T model)
    • SeamlessM4Tv2Config configuration class: SeamlessM4Tv2Model (SeamlessM4Tv2 model)
    • SegGptConfig configuration class: SegGptModel (SegGPT model)
    • SegformerConfig configuration class: SegformerModel (SegFormer model)
    • SiglipConfig configuration class: SiglipModel (SigLIP model)
    • SiglipVisionConfig configuration class: SiglipVisionModel (SiglipVisionModel model)
    • Speech2TextConfig configuration class: Speech2TextModel (Speech2Text model)
    • SpeechT5Config configuration class: SpeechT5Model (SpeechT5 model)
    • SplinterConfig configuration class: SplinterModel (Splinter model)
    • SqueezeBertConfig configuration class: SqueezeBertModel (SqueezeBERT model)
    • StableLmConfig configuration class: StableLmModel (StableLm model)
    • Starcoder2Config configuration class: Starcoder2Model (Starcoder2 model)
    • SwiftFormerConfig configuration class: SwiftFormerModel (SwiftFormer model)
    • Swin2SRConfig configuration class: Swin2SRModel (Swin2SR model)
    • SwinConfig configuration class: SwinModel (Swin Transformer model)
    • Swinv2Config configuration class: Swinv2Model (Swin Transformer V2 model)
    • SwitchTransformersConfig configuration class: SwitchTransformersModel (SwitchTransformers model)
    • T5Config configuration class: T5Model (T5 model)
    • TableTransformerConfig configuration class: TableTransformerModel (Table Transformer model)
    • TapasConfig configuration class: TapasModel (TAPAS model)
    • TimeSeriesTransformerConfig configuration class: TimeSeriesTransformerModel (Time Series Transformer model)
    • TimesformerConfig configuration class: TimesformerModel (TimeSformer model)
    • TimmBackboneConfig configuration class: TimmBackbone (TimmBackbone model)
    • TrajectoryTransformerConfig configuration class: TrajectoryTransformerModel (Trajectory Transformer model)
    • TransfoXLConfig configuration class: TransfoXLModel (Transformer-XL model)
    • TvltConfig configuration class: TvltModel (TVLT model)
    • TvpConfig configuration class: TvpModel (TVP model)
    • UMT5Config configuration class: UMT5Model (UMT5 model)
    • UdopConfig configuration class: UdopModel (UDOP model)
    • UniSpeechConfig configuration class: UniSpeechModel (UniSpeech model)
    • UniSpeechSatConfig configuration class: UniSpeechSatModel (UniSpeechSat model)
    • UnivNetConfig configuration class: UnivNetModel (UnivNet model)
    • VanConfig configuration class: VanModel (VAN model)
    • ViTConfig configuration class: ViTModel (ViT model)
    • ViTHybridConfig configuration class: ViTHybridModel (ViT Hybrid model)
    • ViTMAEConfig configuration class: ViTMAEModel (ViTMAE model)
    • ViTMSNConfig configuration class: ViTMSNModel (ViTMSN model)
    • VideoMAEConfig configuration class: VideoMAEModel (VideoMAE model)
    • ViltConfig configuration class: ViltModel (ViLT model)
    • VisionTextDualEncoderConfig configuration class: VisionTextDualEncoderModel (VisionTextDualEncoder model)
    • VisualBertConfig configuration class: VisualBertModel (VisualBERT model)
    • VitDetConfig configuration class: VitDetModel (VitDet model)
    • VitsConfig configuration class: VitsModel (VITS model)
    • VivitConfig configuration class: VivitModel (ViViT model)
    • Wav2Vec2BertConfig configuration class: Wav2Vec2BertModel (Wav2Vec2-BERT model)
    • Wav2Vec2Config configuration class: Wav2Vec2Model (Wav2Vec2 model)
    • Wav2Vec2ConformerConfig configuration class: Wav2Vec2ConformerModel (Wav2Vec2-Conformer model)
    • WavLMConfig configuration class: WavLMModel (WavLM model)
    • WhisperConfig configuration class: WhisperModel (Whisper model)
    • XCLIPConfig configuration class: XCLIPModel (X-CLIP model)
    • XGLMConfig configuration class: XGLMModel (XGLM model)
    • XLMConfig configuration class: XLMModel (XLM model)
    • XLMProphetNetConfig configuration class: XLMProphetNetModel (XLM-ProphetNet model)
    • XLMRobertaConfig configuration class: XLMRobertaModel (XLM-RoBERTa model)
    • XLMRobertaXLConfig configuration class: XLMRobertaXLModel (XLM-RoBERTa-XL model)
    • XLNetConfig configuration class: XLNetModel (XLNet model)
    • XmodConfig configuration class: XmodModel (X-MOD model)
    • YolosConfig configuration class: YolosModel (YOLOS model)
    • YosoConfig configuration class: YosoModel (YOSO model)
  • attn_implementation (str, optional) — The attention implementation to use in the model (if relevant). Can be any of "eager" (manual implementation of the attention), "sdpa" (using F.scaled_dot_product_attention), or "flash_attention_2" (using Dao-AILab/flash-attention). By default, if available, SDPA will be used for torch>=2.1.1. The default is otherwise the manual "eager" implementation.

Instantiates one of the base model classes of the library from a configuration.

Note: Loading a model from its configuration file does not load the model weights. It only affects the model’s configuration. Use from_pretrained() to load the model weights.

Examples:

>>> from transformers import AutoConfig, AutoModel

>>> # Download configuration from huggingface.co and cache.
>>> config = AutoConfig.from_pretrained("google-bert/bert-base-cased")
>>> model = AutoModel.from_config(config)

from_pretrained

< >

( *model_args **kwargs )

Parameters

  • pretrained_model_name_or_path (str or os.PathLike) — Can be either:

    • A string, the model id of a pretrained model hosted inside a model repo on huggingface.co.
    • A path to a directory containing model weights saved using save_pretrained(), e.g., ./my_model_directory/.
    • A path or url to a tensorflow index checkpoint file (e.g, ./tf_model/model.ckpt.index). In this case, from_tf should be set to True and a configuration object should be provided as config argument. This loading path is slower than converting the TensorFlow checkpoint in a PyTorch model using the provided conversion scripts and loading the PyTorch model afterwards.
  • model_args (additional positional arguments, optional) — Will be passed along to the underlying model __init__() method.
  • config (PretrainedConfig, optional) — Configuration for the model to use instead of an automatically loaded configuration. Configuration can be automatically loaded when:

    • The model is a model provided by the library (loaded with the model id string of a pretrained model).
    • The model was saved using save_pretrained() and is reloaded by supplying the save directory.
    • The model is loaded by supplying a local directory as pretrained_model_name_or_path and a configuration JSON file named config.json is found in the directory.
  • state_dict (Dict[str, torch.Tensor], optional) — A state dictionary to use instead of a state dictionary loaded from saved weights file.

    This option can be used if you want to create a model from a pretrained configuration but load your own weights. In this case though, you should check if using save_pretrained() and from_pretrained() is not a simpler option.

  • cache_dir (str or os.PathLike, optional) — Path to a directory in which a downloaded pretrained model configuration should be cached if the standard cache should not be used.
  • from_tf (bool, optional, defaults to False) — Load the model weights from a TensorFlow checkpoint save file (see docstring of pretrained_model_name_or_path argument).
  • force_download (bool, optional, defaults to False) — Whether or not to force the (re-)download of the model weights and configuration files, overriding the cached versions if they exist.
  • resume_download (bool, optional, defaults to False) — Whether or not to delete incompletely received files. Will attempt to resume the download if such a file exists.
  • proxies (Dict[str, str], optional) — A dictionary of proxy servers to use by protocol or endpoint, e.g., {'http': 'foo.bar:3128', 'http://hostname': 'foo.bar:4012'}. The proxies are used on each request.
  • output_loading_info(bool, optional, defaults to False) — Whether ot not to also return a dictionary containing missing keys, unexpected keys and error messages.
  • local_files_only(bool, optional, defaults to False) — Whether or not to only look at local files (e.g., not try downloading the model).
  • revision (str, optional, defaults to "main") — The specific model version to use. It can be a branch name, a tag name, or a commit id, since we use a git-based system for storing models and other artifacts on huggingface.co, so revision can be any identifier allowed by git.
  • trust_remote_code (bool, optional, defaults to False) — Whether or not to allow for custom models defined on the Hub in their own modeling files. This option should only be set to True for repositories you trust and in which you have read the code, as it will execute code present on the Hub on your local machine.
  • code_revision (str, optional, defaults to "main") — The specific revision to use for the code on the Hub, if the code leaves in a different repository than the rest of the model. It can be a branch name, a tag name, or a commit id, since we use a git-based system for storing models and other artifacts on huggingface.co, so revision can be any identifier allowed by git.
  • kwargs (additional keyword arguments, optional) — Can be used to update the configuration object (after it being loaded) and initiate the model (e.g., output_attentions=True). Behaves differently depending on whether a config is provided or automatically loaded:

    • If a configuration is provided with config, **kwargs will be directly passed to the underlying model’s __init__ method (we assume all relevant updates to the configuration have already been done)
    • If a configuration is not provided, kwargs will be first passed to the configuration class initialization function (from_pretrained()). Each key of kwargs that corresponds to a configuration attribute will be used to override said attribute with the supplied kwargs value. Remaining keys that do not correspond to any configuration attribute will be passed to the underlying model’s __init__ function.

Instantiate one of the base model classes of the library from a pretrained model.

The model class to instantiate is selected based on the model_type property of the config object (either passed as an argument or loaded from pretrained_model_name_or_path if possible), or when it’s missing, by falling back to using pattern matching on pretrained_model_name_or_path:

  • albert — AlbertModel (ALBERT model)
  • align — AlignModel (ALIGN model)
  • altclip — AltCLIPModel (AltCLIP model)
  • audio-spectrogram-transformer — ASTModel (Audio Spectrogram Transformer model)
  • autoformer — AutoformerModel (Autoformer model)
  • bark — BarkModel (Bark model)
  • bart — BartModel (BART model)
  • beit — BeitModel (BEiT model)
  • bert — BertModel (BERT model)
  • bert-generation — BertGenerationEncoder (Bert Generation model)
  • big_bird — BigBirdModel (BigBird model)
  • bigbird_pegasus — BigBirdPegasusModel (BigBird-Pegasus model)
  • biogpt — BioGptModel (BioGpt model)
  • bit — BitModel (BiT model)
  • blenderbot — BlenderbotModel (Blenderbot model)
  • blenderbot-small — BlenderbotSmallModel (BlenderbotSmall model)
  • blip — BlipModel (BLIP model)
  • blip-2 — Blip2Model (BLIP-2 model)
  • bloom — BloomModel (BLOOM model)
  • bridgetower — BridgeTowerModel (BridgeTower model)
  • bros — BrosModel (BROS model)
  • camembert — CamembertModel (CamemBERT model)
  • canine — CanineModel (CANINE model)
  • chinese_clip — ChineseCLIPModel (Chinese-CLIP model)
  • chinese_clip_vision_model — ChineseCLIPVisionModel (ChineseCLIPVisionModel model)
  • clap — ClapModel (CLAP model)
  • clip — CLIPModel (CLIP model)
  • clip_vision_model — CLIPVisionModel (CLIPVisionModel model)
  • clipseg — CLIPSegModel (CLIPSeg model)
  • clvp — ClvpModelForConditionalGeneration (CLVP model)
  • code_llama — LlamaModel (CodeLlama model)
  • codegen — CodeGenModel (CodeGen model)
  • cohere — CohereModel (Cohere model)
  • conditional_detr — ConditionalDetrModel (Conditional DETR model)
  • convbert — ConvBertModel (ConvBERT model)
  • convnext — ConvNextModel (ConvNeXT model)
  • convnextv2 — ConvNextV2Model (ConvNeXTV2 model)
  • cpmant — CpmAntModel (CPM-Ant model)
  • ctrl — CTRLModel (CTRL model)
  • cvt — CvtModel (CvT model)
  • data2vec-audio — Data2VecAudioModel (Data2VecAudio model)
  • data2vec-text — Data2VecTextModel (Data2VecText model)
  • data2vec-vision — Data2VecVisionModel (Data2VecVision model)
  • dbrx — DbrxModel (DBRX model)
  • deberta — DebertaModel (DeBERTa model)
  • deberta-v2 — DebertaV2Model (DeBERTa-v2 model)
  • decision_transformer — DecisionTransformerModel (Decision Transformer model)
  • deformable_detr — DeformableDetrModel (Deformable DETR model)
  • deit — DeiTModel (DeiT model)
  • deta — DetaModel (DETA model)
  • detr — DetrModel (DETR model)
  • dinat — DinatModel (DiNAT model)
  • dinov2 — Dinov2Model (DINOv2 model)
  • distilbert — DistilBertModel (DistilBERT model)
  • donut-swin — DonutSwinModel (DonutSwin model)
  • dpr — DPRQuestionEncoder (DPR model)
  • dpt — DPTModel (DPT model)
  • efficientformer — EfficientFormerModel (EfficientFormer model)
  • efficientnet — EfficientNetModel (EfficientNet model)
  • electra — ElectraModel (ELECTRA model)
  • encodec — EncodecModel (EnCodec model)
  • ernie — ErnieModel (ERNIE model)
  • ernie_m — ErnieMModel (ErnieM model)
  • esm — EsmModel (ESM model)
  • falcon — FalconModel (Falcon model)
  • fastspeech2_conformer — FastSpeech2ConformerModel (FastSpeech2Conformer model)
  • flaubert — FlaubertModel (FlauBERT model)
  • flava — FlavaModel (FLAVA model)
  • fnet — FNetModel (FNet model)
  • focalnet — FocalNetModel (FocalNet model)
  • fsmt — FSMTModel (FairSeq Machine-Translation model)
  • funnel — FunnelModel or FunnelBaseModel (Funnel Transformer model)
  • gemma — GemmaModel (Gemma model)
  • git — GitModel (GIT model)
  • glpn — GLPNModel (GLPN model)
  • gpt-sw3 — GPT2Model (GPT-Sw3 model)
  • gpt2 — GPT2Model (OpenAI GPT-2 model)
  • gpt_bigcode — GPTBigCodeModel (GPTBigCode model)
  • gpt_neo — GPTNeoModel (GPT Neo model)
  • gpt_neox — GPTNeoXModel (GPT NeoX model)
  • gpt_neox_japanese — GPTNeoXJapaneseModel (GPT NeoX Japanese model)
  • gptj — GPTJModel (GPT-J model)
  • gptsan-japanese — GPTSanJapaneseForConditionalGeneration (GPTSAN-japanese model)
  • graphormer — GraphormerModel (Graphormer model)
  • grounding-dino — GroundingDinoModel (Grounding DINO model)
  • groupvit — GroupViTModel (GroupViT model)
  • hubert — HubertModel (Hubert model)
  • ibert — IBertModel (I-BERT model)
  • idefics — IdeficsModel (IDEFICS model)
  • idefics2 — Idefics2Model (Idefics2 model)
  • imagegpt — ImageGPTModel (ImageGPT model)
  • informer — InformerModel (Informer model)
  • jamba — JambaModel (Jamba model)
  • jukebox — JukeboxModel (Jukebox model)
  • kosmos-2 — Kosmos2Model (KOSMOS-2 model)
  • layoutlm — LayoutLMModel (LayoutLM model)
  • layoutlmv2 — LayoutLMv2Model (LayoutLMv2 model)
  • layoutlmv3 — LayoutLMv3Model (LayoutLMv3 model)
  • led — LEDModel (LED model)
  • levit — LevitModel (LeViT model)
  • lilt — LiltModel (LiLT model)
  • llama — LlamaModel (LLaMA model)
  • longformer — LongformerModel (Longformer model)
  • longt5 — LongT5Model (LongT5 model)
  • luke — LukeModel (LUKE model)
  • lxmert — LxmertModel (LXMERT model)
  • m2m_100 — M2M100Model (M2M100 model)
  • mamba — MambaModel (Mamba model)
  • marian — MarianModel (Marian model)
  • markuplm — MarkupLMModel (MarkupLM model)
  • mask2former — Mask2FormerModel (Mask2Former model)
  • maskformer — MaskFormerModel (MaskFormer model)
  • maskformer-swin — MaskFormerSwinModel (MaskFormerSwin model)
  • mbart — MBartModel (mBART model)
  • mctct — MCTCTModel (M-CTC-T model)
  • mega — MegaModel (MEGA model)
  • megatron-bert — MegatronBertModel (Megatron-BERT model)
  • mgp-str — MgpstrForSceneTextRecognition (MGP-STR model)
  • mistral — MistralModel (Mistral model)
  • mixtral — MixtralModel (Mixtral model)
  • mobilebert — MobileBertModel (MobileBERT model)
  • mobilenet_v1 — MobileNetV1Model (MobileNetV1 model)
  • mobilenet_v2 — MobileNetV2Model (MobileNetV2 model)
  • mobilevit — MobileViTModel (MobileViT model)
  • mobilevitv2 — MobileViTV2Model (MobileViTV2 model)
  • mpnet — MPNetModel (MPNet model)
  • mpt — MptModel (MPT model)
  • mra — MraModel (MRA model)
  • mt5 — MT5Model (MT5 model)
  • mvp — MvpModel (MVP model)
  • nat — NatModel (NAT model)
  • nezha — NezhaModel (Nezha model)
  • nllb-moe — NllbMoeModel (NLLB-MOE model)
  • nystromformer — NystromformerModel (Nyströmformer model)
  • olmo — OlmoModel (OLMo model)
  • oneformer — OneFormerModel (OneFormer model)
  • open-llama — OpenLlamaModel (OpenLlama model)
  • openai-gpt — OpenAIGPTModel (OpenAI GPT model)
  • opt — OPTModel (OPT model)
  • owlv2 — Owlv2Model (OWLv2 model)
  • owlvit — OwlViTModel (OWL-ViT model)
  • patchtsmixer — PatchTSMixerModel (PatchTSMixer model)
  • patchtst — PatchTSTModel (PatchTST model)
  • pegasus — PegasusModel (Pegasus model)
  • pegasus_x — PegasusXModel (PEGASUS-X model)
  • perceiver — PerceiverModel (Perceiver model)
  • persimmon — PersimmonModel (Persimmon model)
  • phi — PhiModel (Phi model)
  • plbart — PLBartModel (PLBart model)
  • poolformer — PoolFormerModel (PoolFormer model)
  • prophetnet — ProphetNetModel (ProphetNet model)
  • pvt — PvtModel (PVT model)
  • pvt_v2 — PvtV2Model (PVTv2 model)
  • qdqbert — QDQBertModel (QDQBert model)
  • qwen2 — Qwen2Model (Qwen2 model)
  • qwen2_moe — Qwen2MoeModel (Qwen2MoE model)
  • recurrent_gemma — RecurrentGemmaModel (RecurrentGemma model)
  • reformer — ReformerModel (Reformer model)
  • regnet — RegNetModel (RegNet model)
  • rembert — RemBertModel (RemBERT model)
  • resnet — ResNetModel (ResNet model)
  • retribert — RetriBertModel (RetriBERT model)
  • roberta — RobertaModel (RoBERTa model)
  • roberta-prelayernorm — RobertaPreLayerNormModel (RoBERTa-PreLayerNorm model)
  • roc_bert — RoCBertModel (RoCBert model)
  • roformer — RoFormerModel (RoFormer model)
  • rwkv — RwkvModel (RWKV model)
  • sam — SamModel (SAM model)
  • seamless_m4t — SeamlessM4TModel (SeamlessM4T model)
  • seamless_m4t_v2 — SeamlessM4Tv2Model (SeamlessM4Tv2 model)
  • segformer — SegformerModel (SegFormer model)
  • seggpt — SegGptModel (SegGPT model)
  • sew — SEWModel (SEW model)
  • sew-d — SEWDModel (SEW-D model)
  • siglip — SiglipModel (SigLIP model)
  • siglip_vision_model — SiglipVisionModel (SiglipVisionModel model)
  • speech_to_text — Speech2TextModel (Speech2Text model)
  • speecht5 — SpeechT5Model (SpeechT5 model)
  • splinter — SplinterModel (Splinter model)
  • squeezebert — SqueezeBertModel (SqueezeBERT model)
  • stablelm — StableLmModel (StableLm model)
  • starcoder2 — Starcoder2Model (Starcoder2 model)
  • swiftformer — SwiftFormerModel (SwiftFormer model)
  • swin — SwinModel (Swin Transformer model)
  • swin2sr — Swin2SRModel (Swin2SR model)
  • swinv2 — Swinv2Model (Swin Transformer V2 model)
  • switch_transformers — SwitchTransformersModel (SwitchTransformers model)
  • t5 — T5Model (T5 model)
  • table-transformer — TableTransformerModel (Table Transformer model)
  • tapas — TapasModel (TAPAS model)
  • time_series_transformer — TimeSeriesTransformerModel (Time Series Transformer model)
  • timesformer — TimesformerModel (TimeSformer model)
  • timm_backbone — TimmBackbone (TimmBackbone model)
  • trajectory_transformer — TrajectoryTransformerModel (Trajectory Transformer model)
  • transfo-xl — TransfoXLModel (Transformer-XL model)
  • tvlt — TvltModel (TVLT model)
  • tvp — TvpModel (TVP model)
  • udop — UdopModel (UDOP model)
  • umt5 — UMT5Model (UMT5 model)
  • unispeech — UniSpeechModel (UniSpeech model)
  • unispeech-sat — UniSpeechSatModel (UniSpeechSat model)
  • univnet — UnivNetModel (UnivNet model)
  • van — VanModel (VAN model)
  • videomae — VideoMAEModel (VideoMAE model)
  • vilt — ViltModel (ViLT model)
  • vision-text-dual-encoder — VisionTextDualEncoderModel (VisionTextDualEncoder model)
  • visual_bert — VisualBertModel (VisualBERT model)
  • vit — ViTModel (ViT model)
  • vit_hybrid — ViTHybridModel (ViT Hybrid model)
  • vit_mae — ViTMAEModel (ViTMAE model)
  • vit_msn — ViTMSNModel (ViTMSN model)
  • vitdet — VitDetModel (VitDet model)
  • vits — VitsModel (VITS model)
  • vivit — VivitModel (ViViT model)
  • wav2vec2 — Wav2Vec2Model (Wav2Vec2 model)
  • wav2vec2-bert — Wav2Vec2BertModel (Wav2Vec2-BERT model)
  • wav2vec2-conformer — Wav2Vec2ConformerModel (Wav2Vec2-Conformer model)
  • wavlm — WavLMModel (WavLM model)
  • whisper — WhisperModel (Whisper model)
  • xclip — XCLIPModel (X-CLIP model)
  • xglm — XGLMModel (XGLM model)
  • xlm — XLMModel (XLM model)
  • xlm-prophetnet — XLMProphetNetModel (XLM-ProphetNet model)
  • xlm-roberta — XLMRobertaModel (XLM-RoBERTa model)
  • xlm-roberta-xl — XLMRobertaXLModel (XLM-RoBERTa-XL model)
  • xlnet — XLNetModel (XLNet model)
  • xmod — XmodModel (X-MOD model)
  • yolos — YolosModel (YOLOS model)
  • yoso — YosoModel (YOSO model)

The model is set in evaluation mode by default using model.eval() (so for instance, dropout modules are deactivated). To train the model, you should first set it back in training mode with model.train()

Examples:

>>> from transformers import AutoConfig, AutoModel

>>> # Download model and configuration from huggingface.co and cache.
>>> model = AutoModel.from_pretrained("google-bert/bert-base-cased")

>>> # Update configuration during loading
>>> model = AutoModel.from_pretrained("google-bert/bert-base-cased", output_attentions=True)
>>> model.config.output_attentions
True

>>> # Loading from a TF checkpoint file instead of a PyTorch model (slower)
>>> config = AutoConfig.from_pretrained("./tf_model/bert_tf_model_config.json")
>>> model = AutoModel.from_pretrained(
...     "./tf_model/bert_tf_checkpoint.ckpt.index", from_tf=True, config=config
... )

TFAutoModel

class transformers.TFAutoModel

< >

( *args **kwargs )

This is a generic model class that will be instantiated as one of the base model classes of the library when created with the from_pretrained() class method or the from_config() class method.

This class cannot be instantiated directly using __init__() (throws an error).

from_config

< >

( **kwargs )

Parameters

  • config (PretrainedConfig) — The model class to instantiate is selected based on the configuration class:

    • AlbertConfig configuration class: TFAlbertModel (ALBERT model)
    • BartConfig configuration class: TFBartModel (BART model)
    • BertConfig configuration class: TFBertModel (BERT model)
    • BlenderbotConfig configuration class: TFBlenderbotModel (Blenderbot model)
    • BlenderbotSmallConfig configuration class: TFBlenderbotSmallModel (BlenderbotSmall model)
    • BlipConfig configuration class: TFBlipModel (BLIP model)
    • CLIPConfig configuration class: TFCLIPModel (CLIP model)
    • CTRLConfig configuration class: TFCTRLModel (CTRL model)
    • CamembertConfig configuration class: TFCamembertModel (CamemBERT model)
    • ConvBertConfig configuration class: TFConvBertModel (ConvBERT model)
    • ConvNextConfig configuration class: TFConvNextModel (ConvNeXT model)
    • ConvNextV2Config configuration class: TFConvNextV2Model (ConvNeXTV2 model)
    • CvtConfig configuration class: TFCvtModel (CvT model)
    • DPRConfig configuration class: TFDPRQuestionEncoder (DPR model)
    • Data2VecVisionConfig configuration class: TFData2VecVisionModel (Data2VecVision model)
    • DebertaConfig configuration class: TFDebertaModel (DeBERTa model)
    • DebertaV2Config configuration class: TFDebertaV2Model (DeBERTa-v2 model)
    • DeiTConfig configuration class: TFDeiTModel (DeiT model)
    • DistilBertConfig configuration class: TFDistilBertModel (DistilBERT model)
    • EfficientFormerConfig configuration class: TFEfficientFormerModel (EfficientFormer model)
    • ElectraConfig configuration class: TFElectraModel (ELECTRA model)
    • EsmConfig configuration class: TFEsmModel (ESM model)
    • FlaubertConfig configuration class: TFFlaubertModel (FlauBERT model)
    • FunnelConfig configuration class: TFFunnelModel or TFFunnelBaseModel (Funnel Transformer model)
    • GPT2Config configuration class: TFGPT2Model (OpenAI GPT-2 model)
    • GPTJConfig configuration class: TFGPTJModel (GPT-J model)
    • GroupViTConfig configuration class: TFGroupViTModel (GroupViT model)
    • HubertConfig configuration class: TFHubertModel (Hubert model)
    • LEDConfig configuration class: TFLEDModel (LED model)
    • LayoutLMConfig configuration class: TFLayoutLMModel (LayoutLM model)
    • LayoutLMv3Config configuration class: TFLayoutLMv3Model (LayoutLMv3 model)
    • LongformerConfig configuration class: TFLongformerModel (Longformer model)
    • LxmertConfig configuration class: TFLxmertModel (LXMERT model)
    • MBartConfig configuration class: TFMBartModel (mBART model)
    • MPNetConfig configuration class: TFMPNetModel (MPNet model)
    • MT5Config configuration class: TFMT5Model (MT5 model)
    • MarianConfig configuration class: TFMarianModel (Marian model)
    • MobileBertConfig configuration class: TFMobileBertModel (MobileBERT model)
    • MobileViTConfig configuration class: TFMobileViTModel (MobileViT model)
    • OPTConfig configuration class: TFOPTModel (OPT model)
    • OpenAIGPTConfig configuration class: TFOpenAIGPTModel (OpenAI GPT model)
    • PegasusConfig configuration class: TFPegasusModel (Pegasus model)
    • RegNetConfig configuration class: TFRegNetModel (RegNet model)
    • RemBertConfig configuration class: TFRemBertModel (RemBERT model)
    • ResNetConfig configuration class: TFResNetModel (ResNet model)
    • RoFormerConfig configuration class: TFRoFormerModel (RoFormer model)
    • RobertaConfig configuration class: TFRobertaModel (RoBERTa model)
    • RobertaPreLayerNormConfig configuration class: TFRobertaPreLayerNormModel (RoBERTa-PreLayerNorm model)
    • SamConfig configuration class: TFSamModel (SAM model)
    • SegformerConfig configuration class: TFSegformerModel (SegFormer model)
    • Speech2TextConfig configuration class: TFSpeech2TextModel (Speech2Text model)
    • SwinConfig configuration class: TFSwinModel (Swin Transformer model)
    • T5Config configuration class: TFT5Model (T5 model)
    • TapasConfig configuration class: TFTapasModel (TAPAS model)
    • TransfoXLConfig configuration class: TFTransfoXLModel (Transformer-XL model)
    • ViTConfig configuration class: TFViTModel (ViT model)
    • ViTMAEConfig configuration class: TFViTMAEModel (ViTMAE model)
    • VisionTextDualEncoderConfig configuration class: TFVisionTextDualEncoderModel (VisionTextDualEncoder model)
    • Wav2Vec2Config configuration class: TFWav2Vec2Model (Wav2Vec2 model)
    • WhisperConfig configuration class: TFWhisperModel (Whisper model)
    • XGLMConfig configuration class: TFXGLMModel (XGLM model)
    • XLMConfig configuration class: TFXLMModel (XLM model)
    • XLMRobertaConfig configuration class: TFXLMRobertaModel (XLM-RoBERTa model)
    • XLNetConfig configuration class: TFXLNetModel (XLNet model)
  • attn_implementation (str, optional) — The attention implementation to use in the model (if relevant). Can be any of "eager" (manual implementation of the attention), "sdpa" (using F.scaled_dot_product_attention), or "flash_attention_2" (using Dao-AILab/flash-attention). By default, if available, SDPA will be used for torch>=2.1.1. The default is otherwise the manual "eager" implementation.

Instantiates one of the base model classes of the library from a configuration.

Note: Loading a model from its configuration file does not load the model weights. It only affects the model’s configuration. Use from_pretrained() to load the model weights.

Examples:

>>> from transformers import AutoConfig, TFAutoModel

>>> # Download configuration from huggingface.co and cache.
>>> config = AutoConfig.from_pretrained("google-bert/bert-base-cased")
>>> model = TFAutoModel.from_config(config)

from_pretrained

< >

( *model_args **kwargs )

Parameters

  • pretrained_model_name_or_path (str or os.PathLike) — Can be either:

    • A string, the model id of a pretrained model hosted inside a model repo on huggingface.co.
    • A path to a directory containing model weights saved using save_pretrained(), e.g., ./my_model_directory/.
    • A path or url to a PyTorch state_dict save file (e.g, ./pt_model/pytorch_model.bin). In this case, from_pt should be set to True and a configuration object should be provided as config argument. This loading path is slower than converting the PyTorch model in a TensorFlow model using the provided conversion scripts and loading the TensorFlow model afterwards.
  • model_args (additional positional arguments, optional) — Will be passed along to the underlying model __init__() method.
  • config (PretrainedConfig, optional) — Configuration for the model to use instead of an automatically loaded configuration. Configuration can be automatically loaded when:

    • The model is a model provided by the library (loaded with the model id string of a pretrained model).
    • The model was saved using save_pretrained() and is reloaded by supplying the save directory.
    • The model is loaded by supplying a local directory as pretrained_model_name_or_path and a configuration JSON file named config.json is found in the directory.
  • cache_dir (str or os.PathLike, optional) — Path to a directory in which a downloaded pretrained model configuration should be cached if the standard cache should not be used.
  • from_pt (bool, optional, defaults to False) — Load the model weights from a PyTorch checkpoint save file (see docstring of pretrained_model_name_or_path argument).
  • force_download (bool, optional, defaults to False) — Whether or not to force the (re-)download of the model weights and configuration files, overriding the cached versions if they exist.
  • resume_download (bool, optional, defaults to False) — Whether or not to delete incompletely received files. Will attempt to resume the download if such a file exists.
  • proxies (Dict[str, str], optional) — A dictionary of proxy servers to use by protocol or endpoint, e.g., {'http': 'foo.bar:3128', 'http://hostname': 'foo.bar:4012'}. The proxies are used on each request.
  • output_loading_info(bool, optional, defaults to False) — Whether ot not to also return a dictionary containing missing keys, unexpected keys and error messages.
  • local_files_only(bool, optional, defaults to False) — Whether or not to only look at local files (e.g., not try downloading the model).
  • revision (str, optional, defaults to "main") — The specific model version to use. It can be a branch name, a tag name, or a commit id, since we use a git-based system for storing models and other artifacts on huggingface.co, so revision can be any identifier allowed by git.
  • trust_remote_code (bool, optional, defaults to False) — Whether or not to allow for custom models defined on the Hub in their own modeling files. This option should only be set to True for repositories you trust and in which you have read the code, as it will execute code present on the Hub on your local machine.
  • code_revision (str, optional, defaults to "main") — The specific revision to use for the code on the Hub, if the code leaves in a different repository than the rest of the model. It can be a branch name, a tag name, or a commit id, since we use a git-based system for storing models and other artifacts on huggingface.co, so revision can be any identifier allowed by git.
  • kwargs (additional keyword arguments, optional) — Can be used to update the configuration object (after it being loaded) and initiate the model (e.g., output_attentions=True). Behaves differently depending on whether a config is provided or automatically loaded:

    • If a configuration is provided with config, **kwargs will be directly passed to the underlying model’s __init__ method (we assume all relevant updates to the configuration have already been done)
    • If a configuration is not provided, kwargs will be first passed to the configuration class initialization function (from_pretrained()). Each key of kwargs that corresponds to a configuration attribute will be used to override said attribute with the supplied kwargs value. Remaining keys that do not correspond to any configuration attribute will be passed to the underlying model’s __init__ function.

Instantiate one of the base model classes of the library from a pretrained model.

The model class to instantiate is selected based on the model_type property of the config object (either passed as an argument or loaded from pretrained_model_name_or_path if possible), or when it’s missing, by falling back to using pattern matching on pretrained_model_name_or_path:

  • albert — TFAlbertModel (ALBERT model)
  • bart — TFBartModel (BART model)
  • bert — TFBertModel (BERT model)
  • blenderbot — TFBlenderbotModel (Blenderbot model)
  • blenderbot-small — TFBlenderbotSmallModel (BlenderbotSmall model)
  • blip — TFBlipModel (BLIP model)
  • camembert — TFCamembertModel (CamemBERT model)
  • clip — TFCLIPModel (CLIP model)
  • convbert — TFConvBertModel (ConvBERT model)
  • convnext — TFConvNextModel (ConvNeXT model)
  • convnextv2 — TFConvNextV2Model (ConvNeXTV2 model)
  • ctrl — TFCTRLModel (CTRL model)
  • cvt — TFCvtModel (CvT model)
  • data2vec-vision — TFData2VecVisionModel (Data2VecVision model)
  • deberta — TFDebertaModel (DeBERTa model)
  • deberta-v2 — TFDebertaV2Model (DeBERTa-v2 model)
  • deit — TFDeiTModel (DeiT model)
  • distilbert — TFDistilBertModel (DistilBERT model)
  • dpr — TFDPRQuestionEncoder (DPR model)
  • efficientformer — TFEfficientFormerModel (EfficientFormer model)
  • electra — TFElectraModel (ELECTRA model)
  • esm — TFEsmModel (ESM model)
  • flaubert — TFFlaubertModel (FlauBERT model)
  • funnel — TFFunnelModel or TFFunnelBaseModel (Funnel Transformer model)
  • gpt-sw3 — TFGPT2Model (GPT-Sw3 model)
  • gpt2 — TFGPT2Model (OpenAI GPT-2 model)
  • gptj — TFGPTJModel (GPT-J model)
  • groupvit — TFGroupViTModel (GroupViT model)
  • hubert — TFHubertModel (Hubert model)
  • layoutlm — TFLayoutLMModel (LayoutLM model)
  • layoutlmv3 — TFLayoutLMv3Model (LayoutLMv3 model)
  • led — TFLEDModel (LED model)
  • longformer — TFLongformerModel (Longformer model)
  • lxmert — TFLxmertModel (LXMERT model)
  • marian — TFMarianModel (Marian model)
  • mbart — TFMBartModel (mBART model)
  • mobilebert — TFMobileBertModel (MobileBERT model)
  • mobilevit — TFMobileViTModel (MobileViT model)
  • mpnet — TFMPNetModel (MPNet model)
  • mt5 — TFMT5Model (MT5 model)
  • openai-gpt — TFOpenAIGPTModel (OpenAI GPT model)
  • opt — TFOPTModel (OPT model)
  • pegasus — TFPegasusModel (Pegasus model)
  • regnet — TFRegNetModel (RegNet model)
  • rembert — TFRemBertModel (RemBERT model)
  • resnet — TFResNetModel (ResNet model)
  • roberta — TFRobertaModel (RoBERTa model)
  • roberta-prelayernorm — TFRobertaPreLayerNormModel (RoBERTa-PreLayerNorm model)
  • roformer — TFRoFormerModel (RoFormer model)
  • sam — TFSamModel (SAM model)
  • segformer — TFSegformerModel (SegFormer model)
  • speech_to_text — TFSpeech2TextModel (Speech2Text model)
  • swin — TFSwinModel (Swin Transformer model)
  • t5 — TFT5Model (T5 model)
  • tapas — TFTapasModel (TAPAS model)
  • transfo-xl — TFTransfoXLModel (Transformer-XL model)
  • vision-text-dual-encoder — TFVisionTextDualEncoderModel (VisionTextDualEncoder model)
  • vit — TFViTModel (ViT model)
  • vit_mae — TFViTMAEModel (ViTMAE model)
  • wav2vec2 — TFWav2Vec2Model (Wav2Vec2 model)
  • whisper — TFWhisperModel (Whisper model)
  • xglm — TFXGLMModel (XGLM model)
  • xlm — TFXLMModel (XLM model)
  • xlm-roberta — TFXLMRobertaModel (XLM-RoBERTa model)
  • xlnet — TFXLNetModel (XLNet model)

Examples:

>>> from transformers import AutoConfig, TFAutoModel

>>> # Download model and configuration from huggingface.co and cache.
>>> model = TFAutoModel.from_pretrained("google-bert/bert-base-cased")

>>> # Update configuration during loading
>>> model = TFAutoModel.from_pretrained("google-bert/bert-base-cased", output_attentions=True)
>>> model.config.output_attentions
True

>>> # Loading from a PyTorch checkpoint file instead of a TensorFlow model (slower)
>>> config = AutoConfig.from_pretrained("./pt_model/bert_pt_model_config.json")
>>> model = TFAutoModel.from_pretrained(
...     "./pt_model/bert_pytorch_model.bin", from_pt=True, config=config
... )

FlaxAutoModel

class transformers.FlaxAutoModel

< >

( *args **kwargs )

This is a generic model class that will be instantiated as one of the base model classes of the library when created with the from_pretrained() class method or the from_config() class method.

This class cannot be instantiated directly using __init__() (throws an error).

from_config

< >

( **kwargs )

Parameters

  • config (PretrainedConfig) — The model class to instantiate is selected based on the configuration class:

    • AlbertConfig configuration class: FlaxAlbertModel (ALBERT model)
    • BartConfig configuration class: FlaxBartModel (BART model)
    • BeitConfig configuration class: FlaxBeitModel (BEiT model)
    • BertConfig configuration class: FlaxBertModel (BERT model)
    • BigBirdConfig configuration class: FlaxBigBirdModel (BigBird model)
    • BlenderbotConfig configuration class: FlaxBlenderbotModel (Blenderbot model)
    • BlenderbotSmallConfig configuration class: FlaxBlenderbotSmallModel (BlenderbotSmall model)
    • BloomConfig configuration class: FlaxBloomModel (BLOOM model)
    • CLIPConfig configuration class: FlaxCLIPModel (CLIP model)
    • DistilBertConfig configuration class: FlaxDistilBertModel (DistilBERT model)
    • ElectraConfig configuration class: FlaxElectraModel (ELECTRA model)
    • GPT2Config configuration class: FlaxGPT2Model (OpenAI GPT-2 model)
    • GPTJConfig configuration class: FlaxGPTJModel (GPT-J model)
    • GPTNeoConfig configuration class: FlaxGPTNeoModel (GPT Neo model)
    • GemmaConfig configuration class: FlaxGemmaModel (Gemma model)
    • LlamaConfig configuration class: FlaxLlamaModel (LLaMA model)
    • LongT5Config configuration class: FlaxLongT5Model (LongT5 model)
    • MBartConfig configuration class: FlaxMBartModel (mBART model)
    • MT5Config configuration class: FlaxMT5Model (MT5 model)
    • MarianConfig configuration class: FlaxMarianModel (Marian model)
    • MistralConfig configuration class: FlaxMistralModel (Mistral model)
    • OPTConfig configuration class: FlaxOPTModel (OPT model)
    • PegasusConfig configuration class: FlaxPegasusModel (Pegasus model)
    • RegNetConfig configuration class: FlaxRegNetModel (RegNet model)
    • ResNetConfig configuration class: FlaxResNetModel (ResNet model)
    • RoFormerConfig configuration class: FlaxRoFormerModel (RoFormer model)
    • RobertaConfig configuration class: FlaxRobertaModel (RoBERTa model)
    • RobertaPreLayerNormConfig configuration class: FlaxRobertaPreLayerNormModel (RoBERTa-PreLayerNorm model)
    • T5Config configuration class: FlaxT5Model (T5 model)
    • ViTConfig configuration class: FlaxViTModel (ViT model)
    • VisionTextDualEncoderConfig configuration class: FlaxVisionTextDualEncoderModel (VisionTextDualEncoder model)
    • Wav2Vec2Config configuration class: FlaxWav2Vec2Model (Wav2Vec2 model)
    • WhisperConfig configuration class: FlaxWhisperModel (Whisper model)
    • XGLMConfig configuration class: FlaxXGLMModel (XGLM model)
    • XLMRobertaConfig configuration class: FlaxXLMRobertaModel (XLM-RoBERTa model)
  • attn_implementation (str, optional) — The attention implementation to use in the model (if relevant). Can be any of "eager" (manual implementation of the attention), "sdpa" (using F.scaled_dot_product_attention), or "flash_attention_2" (using Dao-AILab/flash-attention). By default, if available, SDPA will be used for torch>=2.1.1. The default is otherwise the manual "eager" implementation.

Instantiates one of the base model classes of the library from a configuration.

Note: Loading a model from its configuration file does not load the model weights. It only affects the model’s configuration. Use from_pretrained() to load the model weights.

Examples:

>>> from transformers import AutoConfig, FlaxAutoModel

>>> # Download configuration from huggingface.co and cache.
>>> config = AutoConfig.from_pretrained("google-bert/bert-base-cased")
>>> model = FlaxAutoModel.from_config(config)

from_pretrained

< >

( *model_args **kwargs )

Parameters

  • pretrained_model_name_or_path (str or os.PathLike) — Can be either:

    • A string, the model id of a pretrained model hosted inside a model repo on huggingface.co.
    • A path to a directory containing model weights saved using save_pretrained(), e.g., ./my_model_directory/.
    • A path or url to a PyTorch state_dict save file (e.g, ./pt_model/pytorch_model.bin). In this case, from_pt should be set to True and a configuration object should be provided as config argument. This loading path is slower than converting the PyTorch model in a TensorFlow model using the provided conversion scripts and loading the TensorFlow model afterwards.
  • model_args (additional positional arguments, optional) — Will be passed along to the underlying model __init__() method.
  • config (PretrainedConfig, optional) — Configuration for the model to use instead of an automatically loaded configuration. Configuration can be automatically loaded when:

    • The model is a model provided by the library (loaded with the model id string of a pretrained model).
    • The model was saved using save_pretrained() and is reloaded by supplying the save directory.
    • The model is loaded by supplying a local directory as pretrained_model_name_or_path and a configuration JSON file named config.json is found in the directory.
  • cache_dir (str or os.PathLike, optional) — Path to a directory in which a downloaded pretrained model configuration should be cached if the standard cache should not be used.
  • from_pt (bool, optional, defaults to False) — Load the model weights from a PyTorch checkpoint save file (see docstring of pretrained_model_name_or_path argument).
  • force_download (bool, optional, defaults to False) — Whether or not to force the (re-)download of the model weights and configuration files, overriding the cached versions if they exist.
  • resume_download (bool, optional, defaults to False) — Whether or not to delete incompletely received files. Will attempt to resume the download if such a file exists.
  • proxies (Dict[str, str], optional) — A dictionary of proxy servers to use by protocol or endpoint, e.g., {'http': 'foo.bar:3128', 'http://hostname': 'foo.bar:4012'}. The proxies are used on each request.
  • output_loading_info(bool, optional, defaults to False) — Whether ot not to also return a dictionary containing missing keys, unexpected keys and error messages.
  • local_files_only(bool, optional, defaults to False) — Whether or not to only look at local files (e.g., not try downloading the model).
  • revision (str, optional, defaults to "main") — The specific model version to use. It can be a branch name, a tag name, or a commit id, since we use a git-based system for storing models and other artifacts on huggingface.co, so revision can be any identifier allowed by git.
  • trust_remote_code (bool, optional, defaults to False) — Whether or not to allow for custom models defined on the Hub in their own modeling files. This option should only be set to True for repositories you trust and in which you have read the code, as it will execute code present on the Hub on your local machine.
  • code_revision (str, optional, defaults to "main") — The specific revision to use for the code on the Hub, if the code leaves in a different repository than the rest of the model. It can be a branch name, a tag name, or a commit id, since we use a git-based system for storing models and other artifacts on huggingface.co, so revision can be any identifier allowed by git.
  • kwargs (additional keyword arguments, optional) — Can be used to update the configuration object (after it being loaded) and initiate the model (e.g., output_attentions=True). Behaves differently depending on whether a config is provided or automatically loaded:

    • If a configuration is provided with config, **kwargs will be directly passed to the underlying model’s __init__ method (we assume all relevant updates to the configuration have already been done)
    • If a configuration is not provided, kwargs will be first passed to the configuration class initialization function (from_pretrained()). Each key of kwargs that corresponds to a configuration attribute will be used to override said attribute with the supplied kwargs value. Remaining keys that do not correspond to any configuration attribute will be passed to the underlying model’s __init__ function.

Instantiate one of the base model classes of the library from a pretrained model.

The model class to instantiate is selected based on the model_type property of the config object (either passed as an argument or loaded from pretrained_model_name_or_path if possible), or when it’s missing, by falling back to using pattern matching on pretrained_model_name_or_path:

  • albert — FlaxAlbertModel (ALBERT model)
  • bart — FlaxBartModel (BART model)
  • beit — FlaxBeitModel (BEiT model)
  • bert — FlaxBertModel (BERT model)
  • big_bird — FlaxBigBirdModel (BigBird model)
  • blenderbot — FlaxBlenderbotModel (Blenderbot model)
  • blenderbot-small — FlaxBlenderbotSmallModel (BlenderbotSmall model)
  • bloom — FlaxBloomModel (BLOOM model)
  • clip — FlaxCLIPModel (CLIP model)
  • distilbert — FlaxDistilBertModel (DistilBERT model)
  • electra — FlaxElectraModel (ELECTRA model)
  • gemma — FlaxGemmaModel (Gemma model)
  • gpt-sw3 — FlaxGPT2Model (GPT-Sw3 model)
  • gpt2 — FlaxGPT2Model (OpenAI GPT-2 model)
  • gpt_neo — FlaxGPTNeoModel (GPT Neo model)
  • gptj — FlaxGPTJModel (GPT-J model)
  • llama — FlaxLlamaModel (LLaMA model)
  • longt5 — FlaxLongT5Model (LongT5 model)
  • marian — FlaxMarianModel (Marian model)
  • mbart — FlaxMBartModel (mBART model)
  • mistral — FlaxMistralModel (Mistral model)
  • mt5 — FlaxMT5Model (MT5 model)
  • opt — FlaxOPTModel (OPT model)
  • pegasus — FlaxPegasusModel (Pegasus model)
  • regnet — FlaxRegNetModel (RegNet model)
  • resnet — FlaxResNetModel (ResNet model)
  • roberta — FlaxRobertaModel (RoBERTa model)
  • roberta-prelayernorm — FlaxRobertaPreLayerNormModel (RoBERTa-PreLayerNorm model)
  • roformer — FlaxRoFormerModel (RoFormer model)
  • t5 — FlaxT5Model (T5 model)
  • vision-text-dual-encoder — FlaxVisionTextDualEncoderModel (VisionTextDualEncoder model)
  • vit — FlaxViTModel (ViT model)
  • wav2vec2 — FlaxWav2Vec2Model (Wav2Vec2 model)
  • whisper — FlaxWhisperModel (Whisper model)
  • xglm — FlaxXGLMModel (XGLM model)
  • xlm-roberta — FlaxXLMRobertaModel (XLM-RoBERTa model)

Examples:

>>> from transformers import AutoConfig, FlaxAutoModel

>>> # Download model and configuration from huggingface.co and cache.
>>> model = FlaxAutoModel.from_pretrained("google-bert/bert-base-cased")

>>> # Update configuration during loading
>>> model = FlaxAutoModel.from_pretrained("google-bert/bert-base-cased", output_attentions=True)
>>> model.config.output_attentions
True

>>> # Loading from a PyTorch checkpoint file instead of a TensorFlow model (slower)
>>> config = AutoConfig.from_pretrained("./pt_model/bert_pt_model_config.json")
>>> model = FlaxAutoModel.from_pretrained(
...     "./pt_model/bert_pytorch_model.bin", from_pt=True, config=config
... )

Generic pretraining classes

以下の自動クラスは、事前学習ヘッドを持つモデルをインスタンス化するために利用可能です。

AutoModelForPreTraining

class transformers.AutoModelForPreTraining

< >

( *args **kwargs )

This is a generic model class that will be instantiated as one of the model classes of the library (with a pretraining head) when created with the from_pretrained() class method or the from_config() class method.

This class cannot be instantiated directly using __init__() (throws an error).

from_config

< >

( **kwargs )

Parameters

  • config (PretrainedConfig) — The model class to instantiate is selected based on the configuration class:

    • AlbertConfig configuration class: AlbertForPreTraining (ALBERT model)
    • BartConfig configuration class: BartForConditionalGeneration (BART model)
    • BertConfig configuration class: BertForPreTraining (BERT model)
    • BigBirdConfig configuration class: BigBirdForPreTraining (BigBird model)
    • BloomConfig configuration class: BloomForCausalLM (BLOOM model)
    • CTRLConfig configuration class: CTRLLMHeadModel (CTRL model)
    • CamembertConfig configuration class: CamembertForMaskedLM (CamemBERT model)
    • Data2VecTextConfig configuration class: Data2VecTextForMaskedLM (Data2VecText model)
    • DebertaConfig configuration class: DebertaForMaskedLM (DeBERTa model)
    • DebertaV2Config configuration class: DebertaV2ForMaskedLM (DeBERTa-v2 model)
    • DistilBertConfig configuration class: DistilBertForMaskedLM (DistilBERT model)
    • ElectraConfig configuration class: ElectraForPreTraining (ELECTRA model)
    • ErnieConfig configuration class: ErnieForPreTraining (ERNIE model)
    • FNetConfig configuration class: FNetForPreTraining (FNet model)
    • FSMTConfig configuration class: FSMTForConditionalGeneration (FairSeq Machine-Translation model)
    • FlaubertConfig configuration class: FlaubertWithLMHeadModel (FlauBERT model)
    • FlavaConfig configuration class: FlavaForPreTraining (FLAVA model)
    • FunnelConfig configuration class: FunnelForPreTraining (Funnel Transformer model)
    • GPT2Config configuration class: GPT2LMHeadModel (OpenAI GPT-2 model)
    • GPTBigCodeConfig configuration class: GPTBigCodeForCausalLM (GPTBigCode model)
    • GPTSanJapaneseConfig configuration class: GPTSanJapaneseForConditionalGeneration (GPTSAN-japanese model)
    • IBertConfig configuration class: IBertForMaskedLM (I-BERT model)
    • Idefics2Config configuration class: Idefics2ForConditionalGeneration (Idefics2 model)
    • IdeficsConfig configuration class: IdeficsForVisionText2Text (IDEFICS model)
    • LayoutLMConfig configuration class: LayoutLMForMaskedLM (LayoutLM model)
    • LlavaConfig configuration class: LlavaForConditionalGeneration (LLaVa model)
    • LlavaNextConfig configuration class: LlavaNextForConditionalGeneration (LLaVA-NeXT model)
    • LongformerConfig configuration class: LongformerForMaskedLM (Longformer model)
    • LukeConfig configuration class: LukeForMaskedLM (LUKE model)
    • LxmertConfig configuration class: LxmertForPreTraining (LXMERT model)
    • MPNetConfig configuration class: MPNetForMaskedLM (MPNet model)
    • MambaConfig configuration class: MambaForCausalLM (Mamba model)
    • MegaConfig configuration class: MegaForMaskedLM (MEGA model)
    • MegatronBertConfig configuration class: MegatronBertForPreTraining (Megatron-BERT model)
    • MobileBertConfig configuration class: MobileBertForPreTraining (MobileBERT model)
    • MptConfig configuration class: MptForCausalLM (MPT model)
    • MraConfig configuration class: MraForMaskedLM (MRA model)
    • MvpConfig configuration class: MvpForConditionalGeneration (MVP model)
    • NezhaConfig configuration class: NezhaForPreTraining (Nezha model)
    • NllbMoeConfig configuration class: NllbMoeForConditionalGeneration (NLLB-MOE model)
    • OpenAIGPTConfig configuration class: OpenAIGPTLMHeadModel (OpenAI GPT model)
    • RetriBertConfig configuration class: RetriBertModel (RetriBERT model)
    • RoCBertConfig configuration class: RoCBertForPreTraining (RoCBert model)
    • RobertaConfig configuration class: RobertaForMaskedLM (RoBERTa model)
    • RobertaPreLayerNormConfig configuration class: RobertaPreLayerNormForMaskedLM (RoBERTa-PreLayerNorm model)
    • RwkvConfig configuration class: RwkvForCausalLM (RWKV model)
    • SplinterConfig configuration class: SplinterForPreTraining (Splinter model)
    • SqueezeBertConfig configuration class: SqueezeBertForMaskedLM (SqueezeBERT model)
    • SwitchTransformersConfig configuration class: SwitchTransformersForConditionalGeneration (SwitchTransformers model)
    • T5Config configuration class: T5ForConditionalGeneration (T5 model)
    • TapasConfig configuration class: TapasForMaskedLM (TAPAS model)
    • TransfoXLConfig configuration class: TransfoXLLMHeadModel (Transformer-XL model)
    • TvltConfig configuration class: TvltForPreTraining (TVLT model)
    • UniSpeechConfig configuration class: UniSpeechForPreTraining (UniSpeech model)
    • UniSpeechSatConfig configuration class: UniSpeechSatForPreTraining (UniSpeechSat model)
    • ViTMAEConfig configuration class: ViTMAEForPreTraining (ViTMAE model)
    • VideoMAEConfig configuration class: VideoMAEForPreTraining (VideoMAE model)
    • VipLlavaConfig configuration class: VipLlavaForConditionalGeneration (VipLlava model)
    • VisualBertConfig configuration class: VisualBertForPreTraining (VisualBERT model)
    • Wav2Vec2Config configuration class: Wav2Vec2ForPreTraining (Wav2Vec2 model)
    • Wav2Vec2ConformerConfig configuration class: Wav2Vec2ConformerForPreTraining (Wav2Vec2-Conformer model)
    • XLMConfig configuration class: XLMWithLMHeadModel (XLM model)
    • XLMRobertaConfig configuration class: XLMRobertaForMaskedLM (XLM-RoBERTa model)
    • XLMRobertaXLConfig configuration class: XLMRobertaXLForMaskedLM (XLM-RoBERTa-XL model)
    • XLNetConfig configuration class: XLNetLMHeadModel (XLNet model)
    • XmodConfig configuration class: XmodForMaskedLM (X-MOD model)
  • attn_implementation (str, optional) — The attention implementation to use in the model (if relevant). Can be any of "eager" (manual implementation of the attention), "sdpa" (using F.scaled_dot_product_attention), or "flash_attention_2" (using Dao-AILab/flash-attention). By default, if available, SDPA will be used for torch>=2.1.1. The default is otherwise the manual "eager" implementation.

Instantiates one of the model classes of the library (with a pretraining head) from a configuration.

Note: Loading a model from its configuration file does not load the model weights. It only affects the model’s configuration. Use from_pretrained() to load the model weights.

Examples:

>>> from transformers import AutoConfig, AutoModelForPreTraining

>>> # Download configuration from huggingface.co and cache.
>>> config = AutoConfig.from_pretrained("google-bert/bert-base-cased")
>>> model = AutoModelForPreTraining.from_config(config)

from_pretrained

< >

( *model_args **kwargs )

Parameters

  • pretrained_model_name_or_path (str or os.PathLike) — Can be either:

    • A string, the model id of a pretrained model hosted inside a model repo on huggingface.co.
    • A path to a directory containing model weights saved using save_pretrained(), e.g., ./my_model_directory/.
    • A path or url to a tensorflow index checkpoint file (e.g, ./tf_model/model.ckpt.index). In this case, from_tf should be set to True and a configuration object should be provided as config argument. This loading path is slower than converting the TensorFlow checkpoint in a PyTorch model using the provided conversion scripts and loading the PyTorch model afterwards.
  • model_args (additional positional arguments, optional) — Will be passed along to the underlying model __init__() method.
  • config (PretrainedConfig, optional) — Configuration for the model to use instead of an automatically loaded configuration. Configuration can be automatically loaded when:

    • The model is a model provided by the library (loaded with the model id string of a pretrained model).
    • The model was saved using save_pretrained() and is reloaded by supplying the save directory.
    • The model is loaded by supplying a local directory as pretrained_model_name_or_path and a configuration JSON file named config.json is found in the directory.
  • state_dict (Dict[str, torch.Tensor], optional) — A state dictionary to use instead of a state dictionary loaded from saved weights file.

    This option can be used if you want to create a model from a pretrained configuration but load your own weights. In this case though, you should check if using save_pretrained() and from_pretrained() is not a simpler option.

  • cache_dir (str or os.PathLike, optional) — Path to a directory in which a downloaded pretrained model configuration should be cached if the standard cache should not be used.
  • from_tf (bool, optional, defaults to False) — Load the model weights from a TensorFlow checkpoint save file (see docstring of pretrained_model_name_or_path argument).
  • force_download (bool, optional, defaults to False) — Whether or not to force the (re-)download of the model weights and configuration files, overriding the cached versions if they exist.
  • resume_download (bool, optional, defaults to False) — Whether or not to delete incompletely received files. Will attempt to resume the download if such a file exists.
  • proxies (Dict[str, str], optional) — A dictionary of proxy servers to use by protocol or endpoint, e.g., {'http': 'foo.bar:3128', 'http://hostname': 'foo.bar:4012'}. The proxies are used on each request.
  • output_loading_info(bool, optional, defaults to False) — Whether ot not to also return a dictionary containing missing keys, unexpected keys and error messages.
  • local_files_only(bool, optional, defaults to False) — Whether or not to only look at local files (e.g., not try downloading the model).
  • revision (str, optional, defaults to "main") — The specific model version to use. It can be a branch name, a tag name, or a commit id, since we use a git-based system for storing models and other artifacts on huggingface.co, so revision can be any identifier allowed by git.
  • trust_remote_code (bool, optional, defaults to False) — Whether or not to allow for custom models defined on the Hub in their own modeling files. This option should only be set to True for repositories you trust and in which you have read the code, as it will execute code present on the Hub on your local machine.
  • code_revision (str, optional, defaults to "main") — The specific revision to use for the code on the Hub, if the code leaves in a different repository than the rest of the model. It can be a branch name, a tag name, or a commit id, since we use a git-based system for storing models and other artifacts on huggingface.co, so revision can be any identifier allowed by git.
  • kwargs (additional keyword arguments, optional) — Can be used to update the configuration object (after it being loaded) and initiate the model (e.g., output_attentions=True). Behaves differently depending on whether a config is provided or automatically loaded:

    • If a configuration is provided with config, **kwargs will be directly passed to the underlying model’s __init__ method (we assume all relevant updates to the configuration have already been done)
    • If a configuration is not provided, kwargs will be first passed to the configuration class initialization function (from_pretrained()). Each key of kwargs that corresponds to a configuration attribute will be used to override said attribute with the supplied kwargs value. Remaining keys that do not correspond to any configuration attribute will be passed to the underlying model’s __init__ function.

Instantiate one of the model classes of the library (with a pretraining head) from a pretrained model.

The model class to instantiate is selected based on the model_type property of the config object (either passed as an argument or loaded from pretrained_model_name_or_path if possible), or when it’s missing, by falling back to using pattern matching on pretrained_model_name_or_path:

  • albert — AlbertForPreTraining (ALBERT model)
  • bart — BartForConditionalGeneration (BART model)
  • bert — BertForPreTraining (BERT model)
  • big_bird — BigBirdForPreTraining (BigBird model)
  • bloom — BloomForCausalLM (BLOOM model)
  • camembert — CamembertForMaskedLM (CamemBERT model)
  • ctrl — CTRLLMHeadModel (CTRL model)
  • data2vec-text — Data2VecTextForMaskedLM (Data2VecText model)
  • deberta — DebertaForMaskedLM (DeBERTa model)
  • deberta-v2 — DebertaV2ForMaskedLM (DeBERTa-v2 model)
  • distilbert — DistilBertForMaskedLM (DistilBERT model)
  • electra — ElectraForPreTraining (ELECTRA model)
  • ernie — ErnieForPreTraining (ERNIE model)
  • flaubert — FlaubertWithLMHeadModel (FlauBERT model)
  • flava — FlavaForPreTraining (FLAVA model)
  • fnet — FNetForPreTraining (FNet model)
  • fsmt — FSMTForConditionalGeneration (FairSeq Machine-Translation model)
  • funnel — FunnelForPreTraining (Funnel Transformer model)
  • gpt-sw3 — GPT2LMHeadModel (GPT-Sw3 model)
  • gpt2 — GPT2LMHeadModel (OpenAI GPT-2 model)
  • gpt_bigcode — GPTBigCodeForCausalLM (GPTBigCode model)
  • gptsan-japanese — GPTSanJapaneseForConditionalGeneration (GPTSAN-japanese model)
  • ibert — IBertForMaskedLM (I-BERT model)
  • idefics — IdeficsForVisionText2Text (IDEFICS model)
  • idefics2 — Idefics2ForConditionalGeneration (Idefics2 model)
  • layoutlm — LayoutLMForMaskedLM (LayoutLM model)
  • llava — LlavaForConditionalGeneration (LLaVa model)
  • llava_next — LlavaNextForConditionalGeneration (LLaVA-NeXT model)
  • longformer — LongformerForMaskedLM (Longformer model)
  • luke — LukeForMaskedLM (LUKE model)
  • lxmert — LxmertForPreTraining (LXMERT model)
  • mamba — MambaForCausalLM (Mamba model)
  • mega — MegaForMaskedLM (MEGA model)
  • megatron-bert — MegatronBertForPreTraining (Megatron-BERT model)
  • mobilebert — MobileBertForPreTraining (MobileBERT model)
  • mpnet — MPNetForMaskedLM (MPNet model)
  • mpt — MptForCausalLM (MPT model)
  • mra — MraForMaskedLM (MRA model)
  • mvp — MvpForConditionalGeneration (MVP model)
  • nezha — NezhaForPreTraining (Nezha model)
  • nllb-moe — NllbMoeForConditionalGeneration (NLLB-MOE model)
  • openai-gpt — OpenAIGPTLMHeadModel (OpenAI GPT model)
  • retribert — RetriBertModel (RetriBERT model)
  • roberta — RobertaForMaskedLM (RoBERTa model)
  • roberta-prelayernorm — RobertaPreLayerNormForMaskedLM (RoBERTa-PreLayerNorm model)
  • roc_bert — RoCBertForPreTraining (RoCBert model)
  • rwkv — RwkvForCausalLM (RWKV model)
  • splinter — SplinterForPreTraining (Splinter model)
  • squeezebert — SqueezeBertForMaskedLM (SqueezeBERT model)
  • switch_transformers — SwitchTransformersForConditionalGeneration (SwitchTransformers model)
  • t5 — T5ForConditionalGeneration (T5 model)
  • tapas — TapasForMaskedLM (TAPAS model)
  • transfo-xl — TransfoXLLMHeadModel (Transformer-XL model)
  • tvlt — TvltForPreTraining (TVLT model)
  • unispeech — UniSpeechForPreTraining (UniSpeech model)
  • unispeech-sat — UniSpeechSatForPreTraining (UniSpeechSat model)
  • videomae — VideoMAEForPreTraining (VideoMAE model)
  • vipllava — VipLlavaForConditionalGeneration (VipLlava model)
  • visual_bert — VisualBertForPreTraining (VisualBERT model)
  • vit_mae — ViTMAEForPreTraining (ViTMAE model)
  • wav2vec2 — Wav2Vec2ForPreTraining (Wav2Vec2 model)
  • wav2vec2-conformer — Wav2Vec2ConformerForPreTraining (Wav2Vec2-Conformer model)
  • xlm — XLMWithLMHeadModel (XLM model)
  • xlm-roberta — XLMRobertaForMaskedLM (XLM-RoBERTa model)
  • xlm-roberta-xl — XLMRobertaXLForMaskedLM (XLM-RoBERTa-XL model)
  • xlnet — XLNetLMHeadModel (XLNet model)
  • xmod — XmodForMaskedLM (X-MOD model)

The model is set in evaluation mode by default using model.eval() (so for instance, dropout modules are deactivated). To train the model, you should first set it back in training mode with model.train()

Examples:

>>> from transformers import AutoConfig, AutoModelForPreTraining

>>> # Download model and configuration from huggingface.co and cache.
>>> model = AutoModelForPreTraining.from_pretrained("google-bert/bert-base-cased")

>>> # Update configuration during loading
>>> model = AutoModelForPreTraining.from_pretrained("google-bert/bert-base-cased", output_attentions=True)
>>> model.config.output_attentions
True

>>> # Loading from a TF checkpoint file instead of a PyTorch model (slower)
>>> config = AutoConfig.from_pretrained("./tf_model/bert_tf_model_config.json")
>>> model = AutoModelForPreTraining.from_pretrained(
...     "./tf_model/bert_tf_checkpoint.ckpt.index", from_tf=True, config=config
... )

TFAutoModelForPreTraining

class transformers.TFAutoModelForPreTraining

< >

( *args **kwargs )

This is a generic model class that will be instantiated as one of the model classes of the library (with a pretraining head) when created with the from_pretrained() class method or the from_config() class method.

This class cannot be instantiated directly using __init__() (throws an error).

from_config

< >

( **kwargs )

Parameters

  • config (PretrainedConfig) — The model class to instantiate is selected based on the configuration class:

    • AlbertConfig configuration class: TFAlbertForPreTraining (ALBERT model)
    • BartConfig configuration class: TFBartForConditionalGeneration (BART model)
    • BertConfig configuration class: TFBertForPreTraining (BERT model)
    • CTRLConfig configuration class: TFCTRLLMHeadModel (CTRL model)
    • CamembertConfig configuration class: TFCamembertForMaskedLM (CamemBERT model)
    • DistilBertConfig configuration class: TFDistilBertForMaskedLM (DistilBERT model)
    • ElectraConfig configuration class: TFElectraForPreTraining (ELECTRA model)
    • FlaubertConfig configuration class: TFFlaubertWithLMHeadModel (FlauBERT model)
    • FunnelConfig configuration class: TFFunnelForPreTraining (Funnel Transformer model)
    • GPT2Config configuration class: TFGPT2LMHeadModel (OpenAI GPT-2 model)
    • LayoutLMConfig configuration class: TFLayoutLMForMaskedLM (LayoutLM model)
    • LxmertConfig configuration class: TFLxmertForPreTraining (LXMERT model)
    • MPNetConfig configuration class: TFMPNetForMaskedLM (MPNet model)
    • MobileBertConfig configuration class: TFMobileBertForPreTraining (MobileBERT model)
    • OpenAIGPTConfig configuration class: TFOpenAIGPTLMHeadModel (OpenAI GPT model)
    • RobertaConfig configuration class: TFRobertaForMaskedLM (RoBERTa model)
    • RobertaPreLayerNormConfig configuration class: TFRobertaPreLayerNormForMaskedLM (RoBERTa-PreLayerNorm model)
    • T5Config configuration class: TFT5ForConditionalGeneration (T5 model)
    • TapasConfig configuration class: TFTapasForMaskedLM (TAPAS model)
    • TransfoXLConfig configuration class: TFTransfoXLLMHeadModel (Transformer-XL model)
    • ViTMAEConfig configuration class: TFViTMAEForPreTraining (ViTMAE model)
    • XLMConfig configuration class: TFXLMWithLMHeadModel (XLM model)
    • XLMRobertaConfig configuration class: TFXLMRobertaForMaskedLM (XLM-RoBERTa model)
    • XLNetConfig configuration class: TFXLNetLMHeadModel (XLNet model)
  • attn_implementation (str, optional) — The attention implementation to use in the model (if relevant). Can be any of "eager" (manual implementation of the attention), "sdpa" (using F.scaled_dot_product_attention), or "flash_attention_2" (using Dao-AILab/flash-attention). By default, if available, SDPA will be used for torch>=2.1.1. The default is otherwise the manual "eager" implementation.

Instantiates one of the model classes of the library (with a pretraining head) from a configuration.

Note: Loading a model from its configuration file does not load the model weights. It only affects the model’s configuration. Use from_pretrained() to load the model weights.

Examples:

>>> from transformers import AutoConfig, TFAutoModelForPreTraining

>>> # Download configuration from huggingface.co and cache.
>>> config = AutoConfig.from_pretrained("google-bert/bert-base-cased")
>>> model = TFAutoModelForPreTraining.from_config(config)

from_pretrained

< >

( *model_args **kwargs )

Parameters

  • pretrained_model_name_or_path (str or os.PathLike) — Can be either:

    • A string, the model id of a pretrained model hosted inside a model repo on huggingface.co.
    • A path to a directory containing model weights saved using save_pretrained(), e.g., ./my_model_directory/.
    • A path or url to a PyTorch state_dict save file (e.g, ./pt_model/pytorch_model.bin). In this case, from_pt should be set to True and a configuration object should be provided as config argument. This loading path is slower than converting the PyTorch model in a TensorFlow model using the provided conversion scripts and loading the TensorFlow model afterwards.
  • model_args (additional positional arguments, optional) — Will be passed along to the underlying model __init__() method.
  • config (PretrainedConfig, optional) — Configuration for the model to use instead of an automatically loaded configuration. Configuration can be automatically loaded when:

    • The model is a model provided by the library (loaded with the model id string of a pretrained model).
    • The model was saved using save_pretrained() and is reloaded by supplying the save directory.
    • The model is loaded by supplying a local directory as pretrained_model_name_or_path and a configuration JSON file named config.json is found in the directory.
  • cache_dir (str or os.PathLike, optional) — Path to a directory in which a downloaded pretrained model configuration should be cached if the standard cache should not be used.
  • from_pt (bool, optional, defaults to False) — Load the model weights from a PyTorch checkpoint save file (see docstring of pretrained_model_name_or_path argument).
  • force_download (bool, optional, defaults to False) — Whether or not to force the (re-)download of the model weights and configuration files, overriding the cached versions if they exist.
  • resume_download (bool, optional, defaults to False) — Whether or not to delete incompletely received files. Will attempt to resume the download if such a file exists.
  • proxies (Dict[str, str], optional) — A dictionary of proxy servers to use by protocol or endpoint, e.g., {'http': 'foo.bar:3128', 'http://hostname': 'foo.bar:4012'}. The proxies are used on each request.
  • output_loading_info(bool, optional, defaults to False) — Whether ot not to also return a dictionary containing missing keys, unexpected keys and error messages.
  • local_files_only(bool, optional, defaults to False) — Whether or not to only look at local files (e.g., not try downloading the model).
  • revision (str, optional, defaults to "main") — The specific model version to use. It can be a branch name, a tag name, or a commit id, since we use a git-based system for storing models and other artifacts on huggingface.co, so revision can be any identifier allowed by git.
  • trust_remote_code (bool, optional, defaults to False) — Whether or not to allow for custom models defined on the Hub in their own modeling files. This option should only be set to True for repositories you trust and in which you have read the code, as it will execute code present on the Hub on your local machine.
  • code_revision (str, optional, defaults to "main") — The specific revision to use for the code on the Hub, if the code leaves in a different repository than the rest of the model. It can be a branch name, a tag name, or a commit id, since we use a git-based system for storing models and other artifacts on huggingface.co, so revision can be any identifier allowed by git.
  • kwargs (additional keyword arguments, optional) — Can be used to update the configuration object (after it being loaded) and initiate the model (e.g., output_attentions=True). Behaves differently depending on whether a config is provided or automatically loaded:

    • If a configuration is provided with config, **kwargs will be directly passed to the underlying model’s __init__ method (we assume all relevant updates to the configuration have already been done)
    • If a configuration is not provided, kwargs will be first passed to the configuration class initialization function (from_pretrained()). Each key of kwargs that corresponds to a configuration attribute will be used to override said attribute with the supplied kwargs value. Remaining keys that do not correspond to any configuration attribute will be passed to the underlying model’s __init__ function.

Instantiate one of the model classes of the library (with a pretraining head) from a pretrained model.

The model class to instantiate is selected based on the model_type property of the config object (either passed as an argument or loaded from pretrained_model_name_or_path if possible), or when it’s missing, by falling back to using pattern matching on pretrained_model_name_or_path:

  • albert — TFAlbertForPreTraining (ALBERT model)
  • bart — TFBartForConditionalGeneration (BART model)
  • bert — TFBertForPreTraining (BERT model)
  • camembert — TFCamembertForMaskedLM (CamemBERT model)
  • ctrl — TFCTRLLMHeadModel (CTRL model)
  • distilbert — TFDistilBertForMaskedLM (DistilBERT model)
  • electra — TFElectraForPreTraining (ELECTRA model)
  • flaubert — TFFlaubertWithLMHeadModel (FlauBERT model)
  • funnel — TFFunnelForPreTraining (Funnel Transformer model)
  • gpt-sw3 — TFGPT2LMHeadModel (GPT-Sw3 model)
  • gpt2 — TFGPT2LMHeadModel (OpenAI GPT-2 model)
  • layoutlm — TFLayoutLMForMaskedLM (LayoutLM model)
  • lxmert — TFLxmertForPreTraining (LXMERT model)
  • mobilebert — TFMobileBertForPreTraining (MobileBERT model)
  • mpnet — TFMPNetForMaskedLM (MPNet model)
  • openai-gpt — TFOpenAIGPTLMHeadModel (OpenAI GPT model)
  • roberta — TFRobertaForMaskedLM (RoBERTa model)
  • roberta-prelayernorm — TFRobertaPreLayerNormForMaskedLM (RoBERTa-PreLayerNorm model)
  • t5 — TFT5ForConditionalGeneration (T5 model)
  • tapas — TFTapasForMaskedLM (TAPAS model)
  • transfo-xl — TFTransfoXLLMHeadModel (Transformer-XL model)
  • vit_mae — TFViTMAEForPreTraining (ViTMAE model)
  • xlm — TFXLMWithLMHeadModel (XLM model)
  • xlm-roberta — TFXLMRobertaForMaskedLM (XLM-RoBERTa model)
  • xlnet — TFXLNetLMHeadModel (XLNet model)

Examples:

>>> from transformers import AutoConfig, TFAutoModelForPreTraining

>>> # Download model and configuration from huggingface.co and cache.
>>> model = TFAutoModelForPreTraining.from_pretrained("google-bert/bert-base-cased")

>>> # Update configuration during loading
>>> model = TFAutoModelForPreTraining.from_pretrained("google-bert/bert-base-cased", output_attentions=True)
>>> model.config.output_attentions
True

>>> # Loading from a PyTorch checkpoint file instead of a TensorFlow model (slower)
>>> config = AutoConfig.from_pretrained("./pt_model/bert_pt_model_config.json")
>>> model = TFAutoModelForPreTraining.from_pretrained(
...     "./pt_model/bert_pytorch_model.bin", from_pt=True, config=config
... )

FlaxAutoModelForPreTraining

class transformers.FlaxAutoModelForPreTraining

< >

( *args **kwargs )

This is a generic model class that will be instantiated as one of the model classes of the library (with a pretraining head) when created with the from_pretrained() class method or the from_config() class method.

This class cannot be instantiated directly using __init__() (throws an error).

from_config

< >

( **kwargs )

Parameters

  • config (PretrainedConfig) — The model class to instantiate is selected based on the configuration class:

    • AlbertConfig configuration class: FlaxAlbertForPreTraining (ALBERT model)
    • BartConfig configuration class: FlaxBartForConditionalGeneration (BART model)
    • BertConfig configuration class: FlaxBertForPreTraining (BERT model)
    • BigBirdConfig configuration class: FlaxBigBirdForPreTraining (BigBird model)
    • ElectraConfig configuration class: FlaxElectraForPreTraining (ELECTRA model)
    • LongT5Config configuration class: FlaxLongT5ForConditionalGeneration (LongT5 model)
    • MBartConfig configuration class: FlaxMBartForConditionalGeneration (mBART model)
    • MT5Config configuration class: FlaxMT5ForConditionalGeneration (MT5 model)
    • RoFormerConfig configuration class: FlaxRoFormerForMaskedLM (RoFormer model)
    • RobertaConfig configuration class: FlaxRobertaForMaskedLM (RoBERTa model)
    • RobertaPreLayerNormConfig configuration class: FlaxRobertaPreLayerNormForMaskedLM (RoBERTa-PreLayerNorm model)
    • T5Config configuration class: FlaxT5ForConditionalGeneration (T5 model)
    • Wav2Vec2Config configuration class: FlaxWav2Vec2ForPreTraining (Wav2Vec2 model)
    • WhisperConfig configuration class: FlaxWhisperForConditionalGeneration (Whisper model)
    • XLMRobertaConfig configuration class: FlaxXLMRobertaForMaskedLM (XLM-RoBERTa model)
  • attn_implementation (str, optional) — The attention implementation to use in the model (if relevant). Can be any of "eager" (manual implementation of the attention), "sdpa" (using F.scaled_dot_product_attention), or "flash_attention_2" (using Dao-AILab/flash-attention). By default, if available, SDPA will be used for torch>=2.1.1. The default is otherwise the manual "eager" implementation.

Instantiates one of the model classes of the library (with a pretraining head) from a configuration.

Note: Loading a model from its configuration file does not load the model weights. It only affects the model’s configuration. Use from_pretrained() to load the model weights.

Examples:

>>> from transformers import AutoConfig, FlaxAutoModelForPreTraining

>>> # Download configuration from huggingface.co and cache.
>>> config = AutoConfig.from_pretrained("google-bert/bert-base-cased")
>>> model = FlaxAutoModelForPreTraining.from_config(config)

from_pretrained

< >

( *model_args **kwargs )

Parameters

  • pretrained_model_name_or_path (str or os.PathLike) — Can be either:

    • A string, the model id of a pretrained model hosted inside a model repo on huggingface.co.
    • A path to a directory containing model weights saved using save_pretrained(), e.g., ./my_model_directory/.
    • A path or url to a PyTorch state_dict save file (e.g, ./pt_model/pytorch_model.bin). In this case, from_pt should be set to True and a configuration object should be provided as config argument. This loading path is slower than converting the PyTorch model in a TensorFlow model using the provided conversion scripts and loading the TensorFlow model afterwards.
  • model_args (additional positional arguments, optional) — Will be passed along to the underlying model __init__() method.
  • config (PretrainedConfig, optional) — Configuration for the model to use instead of an automatically loaded configuration. Configuration can be automatically loaded when:

    • The model is a model provided by the library (loaded with the model id string of a pretrained model).
    • The model was saved using save_pretrained() and is reloaded by supplying the save directory.
    • The model is loaded by supplying a local directory as pretrained_model_name_or_path and a configuration JSON file named config.json is found in the directory.
  • cache_dir (str or os.PathLike, optional) — Path to a directory in which a downloaded pretrained model configuration should be cached if the standard cache should not be used.
  • from_pt (bool, optional, defaults to False) — Load the model weights from a PyTorch checkpoint save file (see docstring of pretrained_model_name_or_path argument).
  • force_download (bool, optional, defaults to False) — Whether or not to force the (re-)download of the model weights and configuration files, overriding the cached versions if they exist.
  • resume_download (bool, optional, defaults to False) — Whether or not to delete incompletely received files. Will attempt to resume the download if such a file exists.
  • proxies (Dict[str, str], optional) — A dictionary of proxy servers to use by protocol or endpoint, e.g., {'http': 'foo.bar:3128', 'http://hostname': 'foo.bar:4012'}. The proxies are used on each request.
  • output_loading_info(bool, optional, defaults to False) — Whether ot not to also return a dictionary containing missing keys, unexpected keys and error messages.
  • local_files_only(bool, optional, defaults to False) — Whether or not to only look at local files (e.g., not try downloading the model).
  • revision (str, optional, defaults to "main") — The specific model version to use. It can be a branch name, a tag name, or a commit id, since we use a git-based system for storing models and other artifacts on huggingface.co, so revision can be any identifier allowed by git.
  • trust_remote_code (bool, optional, defaults to False) — Whether or not to allow for custom models defined on the Hub in their own modeling files. This option should only be set to True for repositories you trust and in which you have read the code, as it will execute code present on the Hub on your local machine.
  • code_revision (str, optional, defaults to "main") — The specific revision to use for the code on the Hub, if the code leaves in a different repository than the rest of the model. It can be a branch name, a tag name, or a commit id, since we use a git-based system for storing models and other artifacts on huggingface.co, so revision can be any identifier allowed by git.
  • kwargs (additional keyword arguments, optional) — Can be used to update the configuration object (after it being loaded) and initiate the model (e.g., output_attentions=True). Behaves differently depending on whether a config is provided or automatically loaded:

    • If a configuration is provided with config, **kwargs will be directly passed to the underlying model’s __init__ method (we assume all relevant updates to the configuration have already been done)
    • If a configuration is not provided, kwargs will be first passed to the configuration class initialization function (from_pretrained()). Each key of kwargs that corresponds to a configuration attribute will be used to override said attribute with the supplied kwargs value. Remaining keys that do not correspond to any configuration attribute will be passed to the underlying model’s __init__ function.

Instantiate one of the model classes of the library (with a pretraining head) from a pretrained model.

The model class to instantiate is selected based on the model_type property of the config object (either passed as an argument or loaded from pretrained_model_name_or_path if possible), or when it’s missing, by falling back to using pattern matching on pretrained_model_name_or_path:

  • albert — FlaxAlbertForPreTraining (ALBERT model)
  • bart — FlaxBartForConditionalGeneration (BART model)
  • bert — FlaxBertForPreTraining (BERT model)
  • big_bird — FlaxBigBirdForPreTraining (BigBird model)
  • electra — FlaxElectraForPreTraining (ELECTRA model)
  • longt5 — FlaxLongT5ForConditionalGeneration (LongT5 model)
  • mbart — FlaxMBartForConditionalGeneration (mBART model)
  • mt5 — FlaxMT5ForConditionalGeneration (MT5 model)
  • roberta — FlaxRobertaForMaskedLM (RoBERTa model)
  • roberta-prelayernorm — FlaxRobertaPreLayerNormForMaskedLM (RoBERTa-PreLayerNorm model)
  • roformer — FlaxRoFormerForMaskedLM (RoFormer model)
  • t5 — FlaxT5ForConditionalGeneration (T5 model)
  • wav2vec2 — FlaxWav2Vec2ForPreTraining (Wav2Vec2 model)
  • whisper — FlaxWhisperForConditionalGeneration (Whisper model)
  • xlm-roberta — FlaxXLMRobertaForMaskedLM (XLM-RoBERTa model)

Examples:

>>> from transformers import AutoConfig, FlaxAutoModelForPreTraining

>>> # Download model and configuration from huggingface.co and cache.
>>> model = FlaxAutoModelForPreTraining.from_pretrained("google-bert/bert-base-cased")

>>> # Update configuration during loading
>>> model = FlaxAutoModelForPreTraining.from_pretrained("google-bert/bert-base-cased", output_attentions=True)
>>> model.config.output_attentions
True

>>> # Loading from a PyTorch checkpoint file instead of a TensorFlow model (slower)
>>> config = AutoConfig.from_pretrained("./pt_model/bert_pt_model_config.json")
>>> model = FlaxAutoModelForPreTraining.from_pretrained(
...     "./pt_model/bert_pytorch_model.bin", from_pt=True, config=config
... )

Natural Language Processing

以下の自動クラスは、次の自然言語処理タスクに利用可能です。

AutoModelForCausalLM

class transformers.AutoModelForCausalLM

< >

( *args **kwargs )

This is a generic model class that will be instantiated as one of the model classes of the library (with a causal language modeling head) when created with the from_pretrained() class method or the from_config() class method.

This class cannot be instantiated directly using __init__() (throws an error).

from_config

< >

( **kwargs )

Parameters

  • config (PretrainedConfig) — The model class to instantiate is selected based on the configuration class:

    • BartConfig configuration class: BartForCausalLM (BART model)
    • BertConfig configuration class: BertLMHeadModel (BERT model)
    • BertGenerationConfig configuration class: BertGenerationDecoder (Bert Generation model)
    • BigBirdConfig configuration class: BigBirdForCausalLM (BigBird model)
    • BigBirdPegasusConfig configuration class: BigBirdPegasusForCausalLM (BigBird-Pegasus model)
    • BioGptConfig configuration class: BioGptForCausalLM (BioGpt model)
    • BlenderbotConfig configuration class: BlenderbotForCausalLM (Blenderbot model)
    • BlenderbotSmallConfig configuration class: BlenderbotSmallForCausalLM (BlenderbotSmall model)
    • BloomConfig configuration class: BloomForCausalLM (BLOOM model)
    • CTRLConfig configuration class: CTRLLMHeadModel (CTRL model)
    • CamembertConfig configuration class: CamembertForCausalLM (CamemBERT model)
    • CodeGenConfig configuration class: CodeGenForCausalLM (CodeGen model)
    • CohereConfig configuration class: CohereForCausalLM (Cohere model)
    • CpmAntConfig configuration class: CpmAntForCausalLM (CPM-Ant model)
    • Data2VecTextConfig configuration class: Data2VecTextForCausalLM (Data2VecText model)
    • DbrxConfig configuration class: DbrxForCausalLM (DBRX model)
    • ElectraConfig configuration class: ElectraForCausalLM (ELECTRA model)
    • ErnieConfig configuration class: ErnieForCausalLM (ERNIE model)
    • FalconConfig configuration class: FalconForCausalLM (Falcon model)
    • FuyuConfig configuration class: FuyuForCausalLM (Fuyu model)
    • GPT2Config configuration class: GPT2LMHeadModel (OpenAI GPT-2 model)
    • GPTBigCodeConfig configuration class: GPTBigCodeForCausalLM (GPTBigCode model)
    • GPTJConfig configuration class: GPTJForCausalLM (GPT-J model)
    • GPTNeoConfig configuration class: GPTNeoForCausalLM (GPT Neo model)
    • GPTNeoXConfig configuration class: GPTNeoXForCausalLM (GPT NeoX model)
    • GPTNeoXJapaneseConfig configuration class: GPTNeoXJapaneseForCausalLM (GPT NeoX Japanese model)
    • GemmaConfig configuration class: GemmaForCausalLM (Gemma model)
    • GitConfig configuration class: GitForCausalLM (GIT model)
    • JambaConfig configuration class: JambaForCausalLM (Jamba model)
    • LlamaConfig configuration class: LlamaForCausalLM (LLaMA model)
    • MBartConfig configuration class: MBartForCausalLM (mBART model)
    • MambaConfig configuration class: MambaForCausalLM (Mamba model)
    • MarianConfig configuration class: MarianForCausalLM (Marian model)
    • MegaConfig configuration class: MegaForCausalLM (MEGA model)
    • MegatronBertConfig configuration class: MegatronBertForCausalLM (Megatron-BERT model)
    • MistralConfig configuration class: MistralForCausalLM (Mistral model)
    • MixtralConfig configuration class: MixtralForCausalLM (Mixtral model)
    • MptConfig configuration class: MptForCausalLM (MPT model)
    • MusicgenConfig configuration class: MusicgenForCausalLM (MusicGen model)
    • MusicgenMelodyConfig configuration class: MusicgenMelodyForCausalLM (MusicGen Melody model)
    • MvpConfig configuration class: MvpForCausalLM (MVP model)
    • OPTConfig configuration class: OPTForCausalLM (OPT model)
    • OlmoConfig configuration class: OlmoForCausalLM (OLMo model)
    • OpenAIGPTConfig configuration class: OpenAIGPTLMHeadModel (OpenAI GPT model)
    • OpenLlamaConfig configuration class: OpenLlamaForCausalLM (OpenLlama model)
    • PLBartConfig configuration class: PLBartForCausalLM (PLBart model)
    • PegasusConfig configuration class: PegasusForCausalLM (Pegasus model)
    • PersimmonConfig configuration class: PersimmonForCausalLM (Persimmon model)
    • PhiConfig configuration class: PhiForCausalLM (Phi model)
    • ProphetNetConfig configuration class: ProphetNetForCausalLM (ProphetNet model)
    • QDQBertConfig configuration class: QDQBertLMHeadModel (QDQBert model)
    • Qwen2Config configuration class: Qwen2ForCausalLM (Qwen2 model)
    • Qwen2MoeConfig configuration class: Qwen2MoeForCausalLM (Qwen2MoE model)
    • RecurrentGemmaConfig configuration class: RecurrentGemmaForCausalLM (RecurrentGemma model)
    • ReformerConfig configuration class: ReformerModelWithLMHead (Reformer model)
    • RemBertConfig configuration class: RemBertForCausalLM (RemBERT model)
    • RoCBertConfig configuration class: RoCBertForCausalLM (RoCBert model)
    • RoFormerConfig configuration class: RoFormerForCausalLM (RoFormer model)
    • RobertaConfig configuration class: RobertaForCausalLM (RoBERTa model)
    • RobertaPreLayerNormConfig configuration class: RobertaPreLayerNormForCausalLM (RoBERTa-PreLayerNorm model)
    • RwkvConfig configuration class: RwkvForCausalLM (RWKV model)
    • Speech2Text2Config configuration class: Speech2Text2ForCausalLM (Speech2Text2 model)
    • StableLmConfig configuration class: StableLmForCausalLM (StableLm model)
    • Starcoder2Config configuration class: Starcoder2ForCausalLM (Starcoder2 model)
    • TrOCRConfig configuration class: TrOCRForCausalLM (TrOCR model)
    • TransfoXLConfig configuration class: TransfoXLLMHeadModel (Transformer-XL model)
    • WhisperConfig configuration class: WhisperForCausalLM (Whisper model)
    • XGLMConfig configuration class: XGLMForCausalLM (XGLM model)
    • XLMConfig configuration class: XLMWithLMHeadModel (XLM model)
    • XLMProphetNetConfig configuration class: XLMProphetNetForCausalLM (XLM-ProphetNet model)
    • XLMRobertaConfig configuration class: XLMRobertaForCausalLM (XLM-RoBERTa model)
    • XLMRobertaXLConfig configuration class: XLMRobertaXLForCausalLM (XLM-RoBERTa-XL model)
    • XLNetConfig configuration class: XLNetLMHeadModel (XLNet model)
    • XmodConfig configuration class: XmodForCausalLM (X-MOD model)
  • attn_implementation (str, optional) — The attention implementation to use in the model (if relevant). Can be any of "eager" (manual implementation of the attention), "sdpa" (using F.scaled_dot_product_attention), or "flash_attention_2" (using Dao-AILab/flash-attention). By default, if available, SDPA will be used for torch>=2.1.1. The default is otherwise the manual "eager" implementation.

Instantiates one of the model classes of the library (with a causal language modeling head) from a configuration.

Note: Loading a model from its configuration file does not load the model weights. It only affects the model’s configuration. Use from_pretrained() to load the model weights.

Examples:

>>> from transformers import AutoConfig, AutoModelForCausalLM

>>> # Download configuration from huggingface.co and cache.
>>> config = AutoConfig.from_pretrained("google-bert/bert-base-cased")
>>> model = AutoModelForCausalLM.from_config(config)

from_pretrained

< >

( *model_args **kwargs )

Parameters

  • pretrained_model_name_or_path (str or os.PathLike) — Can be either:

    • A string, the model id of a pretrained model hosted inside a model repo on huggingface.co.
    • A path to a directory containing model weights saved using save_pretrained(), e.g., ./my_model_directory/.
    • A path or url to a tensorflow index checkpoint file (e.g, ./tf_model/model.ckpt.index). In this case, from_tf should be set to True and a configuration object should be provided as config argument. This loading path is slower than converting the TensorFlow checkpoint in a PyTorch model using the provided conversion scripts and loading the PyTorch model afterwards.
  • model_args (additional positional arguments, optional) — Will be passed along to the underlying model __init__() method.
  • config (PretrainedConfig, optional) — Configuration for the model to use instead of an automatically loaded configuration. Configuration can be automatically loaded when:

    • The model is a model provided by the library (loaded with the model id string of a pretrained model).
    • The model was saved using save_pretrained() and is reloaded by supplying the save directory.
    • The model is loaded by supplying a local directory as pretrained_model_name_or_path and a configuration JSON file named config.json is found in the directory.
  • state_dict (Dict[str, torch.Tensor], optional) — A state dictionary to use instead of a state dictionary loaded from saved weights file.

    This option can be used if you want to create a model from a pretrained configuration but load your own weights. In this case though, you should check if using save_pretrained() and from_pretrained() is not a simpler option.

  • cache_dir (str or os.PathLike, optional) — Path to a directory in which a downloaded pretrained model configuration should be cached if the standard cache should not be used.
  • from_tf (bool, optional, defaults to False) — Load the model weights from a TensorFlow checkpoint save file (see docstring of pretrained_model_name_or_path argument).
  • force_download (bool, optional, defaults to False) — Whether or not to force the (re-)download of the model weights and configuration files, overriding the cached versions if they exist.
  • resume_download (bool, optional, defaults to False) — Whether or not to delete incompletely received files. Will attempt to resume the download if such a file exists.
  • proxies (Dict[str, str], optional) — A dictionary of proxy servers to use by protocol or endpoint, e.g., {'http': 'foo.bar:3128', 'http://hostname': 'foo.bar:4012'}. The proxies are used on each request.
  • output_loading_info(bool, optional, defaults to False) — Whether ot not to also return a dictionary containing missing keys, unexpected keys and error messages.
  • local_files_only(bool, optional, defaults to False) — Whether or not to only look at local files (e.g., not try downloading the model).
  • revision (str, optional, defaults to "main") — The specific model version to use. It can be a branch name, a tag name, or a commit id, since we use a git-based system for storing models and other artifacts on huggingface.co, so revision can be any identifier allowed by git.
  • trust_remote_code (bool, optional, defaults to False) — Whether or not to allow for custom models defined on the Hub in their own modeling files. This option should only be set to True for repositories you trust and in which you have read the code, as it will execute code present on the Hub on your local machine.
  • code_revision (str, optional, defaults to "main") — The specific revision to use for the code on the Hub, if the code leaves in a different repository than the rest of the model. It can be a branch name, a tag name, or a commit id, since we use a git-based system for storing models and other artifacts on huggingface.co, so revision can be any identifier allowed by git.
  • kwargs (additional keyword arguments, optional) — Can be used to update the configuration object (after it being loaded) and initiate the model (e.g., output_attentions=True). Behaves differently depending on whether a config is provided or automatically loaded:

    • If a configuration is provided with config, **kwargs will be directly passed to the underlying model’s __init__ method (we assume all relevant updates to the configuration have already been done)
    • If a configuration is not provided, kwargs will be first passed to the configuration class initialization function (from_pretrained()). Each key of kwargs that corresponds to a configuration attribute will be used to override said attribute with the supplied kwargs value. Remaining keys that do not correspond to any configuration attribute will be passed to the underlying model’s __init__ function.

Instantiate one of the model classes of the library (with a causal language modeling head) from a pretrained model.

The model class to instantiate is selected based on the model_type property of the config object (either passed as an argument or loaded from pretrained_model_name_or_path if possible), or when it’s missing, by falling back to using pattern matching on pretrained_model_name_or_path:

  • bart — BartForCausalLM (BART model)
  • bert — BertLMHeadModel (BERT model)
  • bert-generation — BertGenerationDecoder (Bert Generation model)
  • big_bird — BigBirdForCausalLM (BigBird model)
  • bigbird_pegasus — BigBirdPegasusForCausalLM (BigBird-Pegasus model)
  • biogpt — BioGptForCausalLM (BioGpt model)
  • blenderbot — BlenderbotForCausalLM (Blenderbot model)
  • blenderbot-small — BlenderbotSmallForCausalLM (BlenderbotSmall model)
  • bloom — BloomForCausalLM (BLOOM model)
  • camembert — CamembertForCausalLM (CamemBERT model)
  • code_llama — LlamaForCausalLM (CodeLlama model)
  • codegen — CodeGenForCausalLM (CodeGen model)
  • cohere — CohereForCausalLM (Cohere model)
  • cpmant — CpmAntForCausalLM (CPM-Ant model)
  • ctrl — CTRLLMHeadModel (CTRL model)
  • data2vec-text — Data2VecTextForCausalLM (Data2VecText model)
  • dbrx — DbrxForCausalLM (DBRX model)
  • electra — ElectraForCausalLM (ELECTRA model)
  • ernie — ErnieForCausalLM (ERNIE model)
  • falcon — FalconForCausalLM (Falcon model)
  • fuyu — FuyuForCausalLM (Fuyu model)
  • gemma — GemmaForCausalLM (Gemma model)
  • git — GitForCausalLM (GIT model)
  • gpt-sw3 — GPT2LMHeadModel (GPT-Sw3 model)
  • gpt2 — GPT2LMHeadModel (OpenAI GPT-2 model)
  • gpt_bigcode — GPTBigCodeForCausalLM (GPTBigCode model)
  • gpt_neo — GPTNeoForCausalLM (GPT Neo model)
  • gpt_neox — GPTNeoXForCausalLM (GPT NeoX model)
  • gpt_neox_japanese — GPTNeoXJapaneseForCausalLM (GPT NeoX Japanese model)
  • gptj — GPTJForCausalLM (GPT-J model)
  • jamba — JambaForCausalLM (Jamba model)
  • llama — LlamaForCausalLM (LLaMA model)
  • mamba — MambaForCausalLM (Mamba model)
  • marian — MarianForCausalLM (Marian model)
  • mbart — MBartForCausalLM (mBART model)
  • mega — MegaForCausalLM (MEGA model)
  • megatron-bert — MegatronBertForCausalLM (Megatron-BERT model)
  • mistral — MistralForCausalLM (Mistral model)
  • mixtral — MixtralForCausalLM (Mixtral model)
  • mpt — MptForCausalLM (MPT model)
  • musicgen — MusicgenForCausalLM (MusicGen model)
  • musicgen_melody — MusicgenMelodyForCausalLM (MusicGen Melody model)
  • mvp — MvpForCausalLM (MVP model)
  • olmo — OlmoForCausalLM (OLMo model)
  • open-llama — OpenLlamaForCausalLM (OpenLlama model)
  • openai-gpt — OpenAIGPTLMHeadModel (OpenAI GPT model)
  • opt — OPTForCausalLM (OPT model)
  • pegasus — PegasusForCausalLM (Pegasus model)
  • persimmon — PersimmonForCausalLM (Persimmon model)
  • phi — PhiForCausalLM (Phi model)
  • plbart — PLBartForCausalLM (PLBart model)
  • prophetnet — ProphetNetForCausalLM (ProphetNet model)
  • qdqbert — QDQBertLMHeadModel (QDQBert model)
  • qwen2 — Qwen2ForCausalLM (Qwen2 model)
  • qwen2_moe — Qwen2MoeForCausalLM (Qwen2MoE model)
  • recurrent_gemma — RecurrentGemmaForCausalLM (RecurrentGemma model)
  • reformer — ReformerModelWithLMHead (Reformer model)
  • rembert — RemBertForCausalLM (RemBERT model)
  • roberta — RobertaForCausalLM (RoBERTa model)
  • roberta-prelayernorm — RobertaPreLayerNormForCausalLM (RoBERTa-PreLayerNorm model)
  • roc_bert — RoCBertForCausalLM (RoCBert model)
  • roformer — RoFormerForCausalLM (RoFormer model)
  • rwkv — RwkvForCausalLM (RWKV model)
  • speech_to_text_2 — Speech2Text2ForCausalLM (Speech2Text2 model)
  • stablelm — StableLmForCausalLM (StableLm model)
  • starcoder2 — Starcoder2ForCausalLM (Starcoder2 model)
  • transfo-xl — TransfoXLLMHeadModel (Transformer-XL model)
  • trocr — TrOCRForCausalLM (TrOCR model)
  • whisper — WhisperForCausalLM (Whisper model)
  • xglm — XGLMForCausalLM (XGLM model)
  • xlm — XLMWithLMHeadModel (XLM model)
  • xlm-prophetnet — XLMProphetNetForCausalLM (XLM-ProphetNet model)
  • xlm-roberta — XLMRobertaForCausalLM (XLM-RoBERTa model)
  • xlm-roberta-xl — XLMRobertaXLForCausalLM (XLM-RoBERTa-XL model)
  • xlnet — XLNetLMHeadModel (XLNet model)
  • xmod — XmodForCausalLM (X-MOD model)

The model is set in evaluation mode by default using model.eval() (so for instance, dropout modules are deactivated). To train the model, you should first set it back in training mode with model.train()

Examples:

>>> from transformers import AutoConfig, AutoModelForCausalLM

>>> # Download model and configuration from huggingface.co and cache.
>>> model = AutoModelForCausalLM.from_pretrained("google-bert/bert-base-cased")

>>> # Update configuration during loading
>>> model = AutoModelForCausalLM.from_pretrained("google-bert/bert-base-cased", output_attentions=True)
>>> model.config.output_attentions
True

>>> # Loading from a TF checkpoint file instead of a PyTorch model (slower)
>>> config = AutoConfig.from_pretrained("./tf_model/bert_tf_model_config.json")
>>> model = AutoModelForCausalLM.from_pretrained(
...     "./tf_model/bert_tf_checkpoint.ckpt.index", from_tf=True, config=config
... )

TFAutoModelForCausalLM

class transformers.TFAutoModelForCausalLM

< >

( *args **kwargs )

This is a generic model class that will be instantiated as one of the model classes of the library (with a causal language modeling head) when created with the from_pretrained() class method or the from_config() class method.

This class cannot be instantiated directly using __init__() (throws an error).

from_config

< >

( **kwargs )

Parameters

  • config (PretrainedConfig) — The model class to instantiate is selected based on the configuration class:

    • BertConfig configuration class: TFBertLMHeadModel (BERT model)
    • CTRLConfig configuration class: TFCTRLLMHeadModel (CTRL model)
    • CamembertConfig configuration class: TFCamembertForCausalLM (CamemBERT model)
    • GPT2Config configuration class: TFGPT2LMHeadModel (OpenAI GPT-2 model)
    • GPTJConfig configuration class: TFGPTJForCausalLM (GPT-J model)
    • OPTConfig configuration class: TFOPTForCausalLM (OPT model)
    • OpenAIGPTConfig configuration class: TFOpenAIGPTLMHeadModel (OpenAI GPT model)
    • RemBertConfig configuration class: TFRemBertForCausalLM (RemBERT model)
    • RoFormerConfig configuration class: TFRoFormerForCausalLM (RoFormer model)
    • RobertaConfig configuration class: TFRobertaForCausalLM (RoBERTa model)
    • RobertaPreLayerNormConfig configuration class: TFRobertaPreLayerNormForCausalLM (RoBERTa-PreLayerNorm model)
    • TransfoXLConfig configuration class: TFTransfoXLLMHeadModel (Transformer-XL model)
    • XGLMConfig configuration class: TFXGLMForCausalLM (XGLM model)
    • XLMConfig configuration class: TFXLMWithLMHeadModel (XLM model)
    • XLMRobertaConfig configuration class: TFXLMRobertaForCausalLM (XLM-RoBERTa model)
    • XLNetConfig configuration class: TFXLNetLMHeadModel (XLNet model)
  • attn_implementation (str, optional) — The attention implementation to use in the model (if relevant). Can be any of "eager" (manual implementation of the attention), "sdpa" (using F.scaled_dot_product_attention), or "flash_attention_2" (using Dao-AILab/flash-attention). By default, if available, SDPA will be used for torch>=2.1.1. The default is otherwise the manual "eager" implementation.

Instantiates one of the model classes of the library (with a causal language modeling head) from a configuration.

Note: Loading a model from its configuration file does not load the model weights. It only affects the model’s configuration. Use from_pretrained() to load the model weights.

Examples:

>>> from transformers import AutoConfig, TFAutoModelForCausalLM

>>> # Download configuration from huggingface.co and cache.
>>> config = AutoConfig.from_pretrained("google-bert/bert-base-cased")
>>> model = TFAutoModelForCausalLM.from_config(config)

from_pretrained

< >

( *model_args **kwargs )

Parameters

  • pretrained_model_name_or_path (str or os.PathLike) — Can be either:

    • A string, the model id of a pretrained model hosted inside a model repo on huggingface.co.
    • A path to a directory containing model weights saved using save_pretrained(), e.g., ./my_model_directory/.
    • A path or url to a PyTorch state_dict save file (e.g, ./pt_model/pytorch_model.bin). In this case, from_pt should be set to True and a configuration object should be provided as config argument. This loading path is slower than converting the PyTorch model in a TensorFlow model using the provided conversion scripts and loading the TensorFlow model afterwards.
  • model_args (additional positional arguments, optional) — Will be passed along to the underlying model __init__() method.
  • config (PretrainedConfig, optional) — Configuration for the model to use instead of an automatically loaded configuration. Configuration can be automatically loaded when:

    • The model is a model provided by the library (loaded with the model id string of a pretrained model).
    • The model was saved using save_pretrained() and is reloaded by supplying the save directory.
    • The model is loaded by supplying a local directory as pretrained_model_name_or_path and a configuration JSON file named config.json is found in the directory.
  • cache_dir (str or os.PathLike, optional) — Path to a directory in which a downloaded pretrained model configuration should be cached if the standard cache should not be used.
  • from_pt (bool, optional, defaults to False) — Load the model weights from a PyTorch checkpoint save file (see docstring of pretrained_model_name_or_path argument).
  • force_download (bool, optional, defaults to False) — Whether or not to force the (re-)download of the model weights and configuration files, overriding the cached versions if they exist.
  • resume_download (bool, optional, defaults to False) — Whether or not to delete incompletely received files. Will attempt to resume the download if such a file exists.
  • proxies (Dict[str, str], optional) — A dictionary of proxy servers to use by protocol or endpoint, e.g., {'http': 'foo.bar:3128', 'http://hostname': 'foo.bar:4012'}. The proxies are used on each request.
  • output_loading_info(bool, optional, defaults to False) — Whether ot not to also return a dictionary containing missing keys, unexpected keys and error messages.
  • local_files_only(bool, optional, defaults to False) — Whether or not to only look at local files (e.g., not try downloading the model).
  • revision (str, optional, defaults to "main") — The specific model version to use. It can be a branch name, a tag name, or a commit id, since we use a git-based system for storing models and other artifacts on huggingface.co, so revision can be any identifier allowed by git.
  • trust_remote_code (bool, optional, defaults to False) — Whether or not to allow for custom models defined on the Hub in their own modeling files. This option should only be set to True for repositories you trust and in which you have read the code, as it will execute code present on the Hub on your local machine.
  • code_revision (str, optional, defaults to "main") — The specific revision to use for the code on the Hub, if the code leaves in a different repository than the rest of the model. It can be a branch name, a tag name, or a commit id, since we use a git-based system for storing models and other artifacts on huggingface.co, so revision can be any identifier allowed by git.
  • kwargs (additional keyword arguments, optional) — Can be used to update the configuration object (after it being loaded) and initiate the model (e.g., output_attentions=True). Behaves differently depending on whether a config is provided or automatically loaded:

    • If a configuration is provided with config, **kwargs will be directly passed to the underlying model’s __init__ method (we assume all relevant updates to the configuration have already been done)
    • If a configuration is not provided, kwargs will be first passed to the configuration class initialization function (from_pretrained()). Each key of kwargs that corresponds to a configuration attribute will be used to override said attribute with the supplied kwargs value. Remaining keys that do not correspond to any configuration attribute will be passed to the underlying model’s __init__ function.

Instantiate one of the model classes of the library (with a causal language modeling head) from a pretrained model.

The model class to instantiate is selected based on the model_type property of the config object (either passed as an argument or loaded from pretrained_model_name_or_path if possible), or when it’s missing, by falling back to using pattern matching on pretrained_model_name_or_path:

  • bert — TFBertLMHeadModel (BERT model)
  • camembert — TFCamembertForCausalLM (CamemBERT model)
  • ctrl — TFCTRLLMHeadModel (CTRL model)
  • gpt-sw3 — TFGPT2LMHeadModel (GPT-Sw3 model)
  • gpt2 — TFGPT2LMHeadModel (OpenAI GPT-2 model)
  • gptj — TFGPTJForCausalLM (GPT-J model)
  • openai-gpt — TFOpenAIGPTLMHeadModel (OpenAI GPT model)
  • opt — TFOPTForCausalLM (OPT model)
  • rembert — TFRemBertForCausalLM (RemBERT model)
  • roberta — TFRobertaForCausalLM (RoBERTa model)
  • roberta-prelayernorm — TFRobertaPreLayerNormForCausalLM (RoBERTa-PreLayerNorm model)
  • roformer — TFRoFormerForCausalLM (RoFormer model)
  • transfo-xl — TFTransfoXLLMHeadModel (Transformer-XL model)
  • xglm — TFXGLMForCausalLM (XGLM model)
  • xlm — TFXLMWithLMHeadModel (XLM model)
  • xlm-roberta — TFXLMRobertaForCausalLM (XLM-RoBERTa model)
  • xlnet — TFXLNetLMHeadModel (XLNet model)

Examples:

>>> from transformers import AutoConfig, TFAutoModelForCausalLM

>>> # Download model and configuration from huggingface.co and cache.
>>> model = TFAutoModelForCausalLM.from_pretrained("google-bert/bert-base-cased")

>>> # Update configuration during loading
>>> model = TFAutoModelForCausalLM.from_pretrained("google-bert/bert-base-cased", output_attentions=True)
>>> model.config.output_attentions
True

>>> # Loading from a PyTorch checkpoint file instead of a TensorFlow model (slower)
>>> config = AutoConfig.from_pretrained("./pt_model/bert_pt_model_config.json")
>>> model = TFAutoModelForCausalLM.from_pretrained(
...     "./pt_model/bert_pytorch_model.bin", from_pt=True, config=config
... )

FlaxAutoModelForCausalLM

class transformers.FlaxAutoModelForCausalLM

< >

( *args **kwargs )

This is a generic model class that will be instantiated as one of the model classes of the library (with a causal language modeling head) when created with the from_pretrained() class method or the from_config() class method.

This class cannot be instantiated directly using __init__() (throws an error).

from_config

< >

( **kwargs )

Parameters

  • config (PretrainedConfig) — The model class to instantiate is selected based on the configuration class:

    • BartConfig configuration class: FlaxBartForCausalLM (BART model)
    • BertConfig configuration class: FlaxBertForCausalLM (BERT model)
    • BigBirdConfig configuration class: FlaxBigBirdForCausalLM (BigBird model)
    • BloomConfig configuration class: FlaxBloomForCausalLM (BLOOM model)
    • ElectraConfig configuration class: FlaxElectraForCausalLM (ELECTRA model)
    • GPT2Config configuration class: FlaxGPT2LMHeadModel (OpenAI GPT-2 model)
    • GPTJConfig configuration class: FlaxGPTJForCausalLM (GPT-J model)
    • GPTNeoConfig configuration class: FlaxGPTNeoForCausalLM (GPT Neo model)
    • GemmaConfig configuration class: FlaxGemmaForCausalLM (Gemma model)
    • LlamaConfig configuration class: FlaxLlamaForCausalLM (LLaMA model)
    • MistralConfig configuration class: FlaxMistralForCausalLM (Mistral model)
    • OPTConfig configuration class: FlaxOPTForCausalLM (OPT model)
    • RobertaConfig configuration class: FlaxRobertaForCausalLM (RoBERTa model)
    • RobertaPreLayerNormConfig configuration class: FlaxRobertaPreLayerNormForCausalLM (RoBERTa-PreLayerNorm model)
    • XGLMConfig configuration class: FlaxXGLMForCausalLM (XGLM model)
    • XLMRobertaConfig configuration class: FlaxXLMRobertaForCausalLM (XLM-RoBERTa model)
  • attn_implementation (str, optional) — The attention implementation to use in the model (if relevant). Can be any of "eager" (manual implementation of the attention), "sdpa" (using F.scaled_dot_product_attention), or "flash_attention_2" (using Dao-AILab/flash-attention). By default, if available, SDPA will be used for torch>=2.1.1. The default is otherwise the manual "eager" implementation.

Instantiates one of the model classes of the library (with a causal language modeling head) from a configuration.

Note: Loading a model from its configuration file does not load the model weights. It only affects the model’s configuration. Use from_pretrained() to load the model weights.

Examples:

>>> from transformers import AutoConfig, FlaxAutoModelForCausalLM

>>> # Download configuration from huggingface.co and cache.
>>> config = AutoConfig.from_pretrained("google-bert/bert-base-cased")
>>> model = FlaxAutoModelForCausalLM.from_config(config)

from_pretrained

< >

( *model_args **kwargs )

Parameters

  • pretrained_model_name_or_path (str or os.PathLike) — Can be either:

    • A string, the model id of a pretrained model hosted inside a model repo on huggingface.co.
    • A path to a directory containing model weights saved using save_pretrained(), e.g., ./my_model_directory/.
    • A path or url to a PyTorch state_dict save file (e.g, ./pt_model/pytorch_model.bin). In this case, from_pt should be set to True and a configuration object should be provided as config argument. This loading path is slower than converting the PyTorch model in a TensorFlow model using the provided conversion scripts and loading the TensorFlow model afterwards.
  • model_args (additional positional arguments, optional) — Will be passed along to the underlying model __init__() method.
  • config (PretrainedConfig, optional) — Configuration for the model to use instead of an automatically loaded configuration. Configuration can be automatically loaded when:

    • The model is a model provided by the library (loaded with the model id string of a pretrained model).
    • The model was saved using save_pretrained() and is reloaded by supplying the save directory.
    • The model is loaded by supplying a local directory as pretrained_model_name_or_path and a configuration JSON file named config.json is found in the directory.
  • cache_dir (str or os.PathLike, optional) — Path to a directory in which a downloaded pretrained model configuration should be cached if the standard cache should not be used.
  • from_pt (bool, optional, defaults to False) — Load the model weights from a PyTorch checkpoint save file (see docstring of pretrained_model_name_or_path argument).
  • force_download (bool, optional, defaults to False) — Whether or not to force the (re-)download of the model weights and configuration files, overriding the cached versions if they exist.
  • resume_download (bool, optional, defaults to False) — Whether or not to delete incompletely received files. Will attempt to resume the download if such a file exists.
  • proxies (Dict[str, str], optional) — A dictionary of proxy servers to use by protocol or endpoint, e.g., {'http': 'foo.bar:3128', 'http://hostname': 'foo.bar:4012'}. The proxies are used on each request.
  • output_loading_info(bool, optional, defaults to False) — Whether ot not to also return a dictionary containing missing keys, unexpected keys and error messages.
  • local_files_only(bool, optional, defaults to False) — Whether or not to only look at local files (e.g., not try downloading the model).
  • revision (str, optional, defaults to "main") — The specific model version to use. It can be a branch name, a tag name, or a commit id, since we use a git-based system for storing models and other artifacts on huggingface.co, so revision can be any identifier allowed by git.
  • trust_remote_code (bool, optional, defaults to False) — Whether or not to allow for custom models defined on the Hub in their own modeling files. This option should only be set to True for repositories you trust and in which you have read the code, as it will execute code present on the Hub on your local machine.
  • code_revision (str, optional, defaults to "main") — The specific revision to use for the code on the Hub, if the code leaves in a different repository than the rest of the model. It can be a branch name, a tag name, or a commit id, since we use a git-based system for storing models and other artifacts on huggingface.co, so revision can be any identifier allowed by git.
  • kwargs (additional keyword arguments, optional) — Can be used to update the configuration object (after it being loaded) and initiate the model (e.g., output_attentions=True). Behaves differently depending on whether a config is provided or automatically loaded:

    • If a configuration is provided with config, **kwargs will be directly passed to the underlying model’s __init__ method (we assume all relevant updates to the configuration have already been done)
    • If a configuration is not provided, kwargs will be first passed to the configuration class initialization function (from_pretrained()). Each key of kwargs that corresponds to a configuration attribute will be used to override said attribute with the supplied kwargs value. Remaining keys that do not correspond to any configuration attribute will be passed to the underlying model’s __init__ function.

Instantiate one of the model classes of the library (with a causal language modeling head) from a pretrained model.

The model class to instantiate is selected based on the model_type property of the config object (either passed as an argument or loaded from pretrained_model_name_or_path if possible), or when it’s missing, by falling back to using pattern matching on pretrained_model_name_or_path:

  • bart — FlaxBartForCausalLM (BART model)
  • bert — FlaxBertForCausalLM (BERT model)
  • big_bird — FlaxBigBirdForCausalLM (BigBird model)
  • bloom — FlaxBloomForCausalLM (BLOOM model)
  • electra — FlaxElectraForCausalLM (ELECTRA model)
  • gemma — FlaxGemmaForCausalLM (Gemma model)
  • gpt-sw3 — FlaxGPT2LMHeadModel (GPT-Sw3 model)
  • gpt2 — FlaxGPT2LMHeadModel (OpenAI GPT-2 model)
  • gpt_neo — FlaxGPTNeoForCausalLM (GPT Neo model)
  • gptj — FlaxGPTJForCausalLM (GPT-J model)
  • llama — FlaxLlamaForCausalLM (LLaMA model)
  • mistral — FlaxMistralForCausalLM (Mistral model)
  • opt — FlaxOPTForCausalLM (OPT model)
  • roberta — FlaxRobertaForCausalLM (RoBERTa model)
  • roberta-prelayernorm — FlaxRobertaPreLayerNormForCausalLM (RoBERTa-PreLayerNorm model)
  • xglm — FlaxXGLMForCausalLM (XGLM model)
  • xlm-roberta — FlaxXLMRobertaForCausalLM (XLM-RoBERTa model)

Examples:

>>> from transformers import AutoConfig, FlaxAutoModelForCausalLM

>>> # Download model and configuration from huggingface.co and cache.
>>> model = FlaxAutoModelForCausalLM.from_pretrained("google-bert/bert-base-cased")

>>> # Update configuration during loading
>>> model = FlaxAutoModelForCausalLM.from_pretrained("google-bert/bert-base-cased", output_attentions=True)
>>> model.config.output_attentions
True

>>> # Loading from a PyTorch checkpoint file instead of a TensorFlow model (slower)
>>> config = AutoConfig.from_pretrained("./pt_model/bert_pt_model_config.json")
>>> model = FlaxAutoModelForCausalLM.from_pretrained(
...     "./pt_model/bert_pytorch_model.bin", from_pt=True, config=config
... )

AutoModelForMaskedLM

class transformers.AutoModelForMaskedLM

< >

( *args **kwargs )

This is a generic model class that will be instantiated as one of the model classes of the library (with a masked language modeling head) when created with the from_pretrained() class method or the from_config() class method.

This class cannot be instantiated directly using __init__() (throws an error).

from_config

< >

( **kwargs )

Parameters

  • config (PretrainedConfig) — The model class to instantiate is selected based on the configuration class:

    • AlbertConfig configuration class: AlbertForMaskedLM (ALBERT model)
    • BartConfig configuration class: BartForConditionalGeneration (BART model)
    • BertConfig configuration class: BertForMaskedLM (BERT model)
    • BigBirdConfig configuration class: BigBirdForMaskedLM (BigBird model)
    • CamembertConfig configuration class: CamembertForMaskedLM (CamemBERT model)
    • ConvBertConfig configuration class: ConvBertForMaskedLM (ConvBERT model)
    • Data2VecTextConfig configuration class: Data2VecTextForMaskedLM (Data2VecText model)
    • DebertaConfig configuration class: DebertaForMaskedLM (DeBERTa model)
    • DebertaV2Config configuration class: DebertaV2ForMaskedLM (DeBERTa-v2 model)
    • DistilBertConfig configuration class: DistilBertForMaskedLM (DistilBERT model)
    • ElectraConfig configuration class: ElectraForMaskedLM (ELECTRA model)
    • ErnieConfig configuration class: ErnieForMaskedLM (ERNIE model)
    • EsmConfig configuration class: EsmForMaskedLM (ESM model)
    • FNetConfig configuration class: FNetForMaskedLM (FNet model)
    • FlaubertConfig configuration class: FlaubertWithLMHeadModel (FlauBERT model)
    • FunnelConfig configuration class: FunnelForMaskedLM (Funnel Transformer model)
    • IBertConfig configuration class: IBertForMaskedLM (I-BERT model)
    • LayoutLMConfig configuration class: LayoutLMForMaskedLM (LayoutLM model)
    • LongformerConfig configuration class: LongformerForMaskedLM (Longformer model)
    • LukeConfig configuration class: LukeForMaskedLM (LUKE model)
    • MBartConfig configuration class: MBartForConditionalGeneration (mBART model)
    • MPNetConfig configuration class: MPNetForMaskedLM (MPNet model)
    • MegaConfig configuration class: MegaForMaskedLM (MEGA model)
    • MegatronBertConfig configuration class: MegatronBertForMaskedLM (Megatron-BERT model)
    • MobileBertConfig configuration class: MobileBertForMaskedLM (MobileBERT model)
    • MraConfig configuration class: MraForMaskedLM (MRA model)
    • MvpConfig configuration class: MvpForConditionalGeneration (MVP model)
    • NezhaConfig configuration class: NezhaForMaskedLM (Nezha model)
    • NystromformerConfig configuration class: NystromformerForMaskedLM (Nyströmformer model)
    • PerceiverConfig configuration class: PerceiverForMaskedLM (Perceiver model)
    • QDQBertConfig configuration class: QDQBertForMaskedLM (QDQBert model)
    • ReformerConfig configuration class: ReformerForMaskedLM (Reformer model)
    • RemBertConfig configuration class: RemBertForMaskedLM (RemBERT model)
    • RoCBertConfig configuration class: RoCBertForMaskedLM (RoCBert model)
    • RoFormerConfig configuration class: RoFormerForMaskedLM (RoFormer model)
    • RobertaConfig configuration class: RobertaForMaskedLM (RoBERTa model)
    • RobertaPreLayerNormConfig configuration class: RobertaPreLayerNormForMaskedLM (RoBERTa-PreLayerNorm model)
    • SqueezeBertConfig configuration class: SqueezeBertForMaskedLM (SqueezeBERT model)
    • TapasConfig configuration class: TapasForMaskedLM (TAPAS model)
    • Wav2Vec2Config configuration class: Wav2Vec2ForMaskedLM (Wav2Vec2 model)
    • XLMConfig configuration class: XLMWithLMHeadModel (XLM model)
    • XLMRobertaConfig configuration class: XLMRobertaForMaskedLM (XLM-RoBERTa model)
    • XLMRobertaXLConfig configuration class: XLMRobertaXLForMaskedLM (XLM-RoBERTa-XL model)
    • XmodConfig configuration class: XmodForMaskedLM (X-MOD model)
    • YosoConfig configuration class: YosoForMaskedLM (YOSO model)
  • attn_implementation (str, optional) — The attention implementation to use in the model (if relevant). Can be any of "eager" (manual implementation of the attention), "sdpa" (using F.scaled_dot_product_attention), or "flash_attention_2" (using Dao-AILab/flash-attention). By default, if available, SDPA will be used for torch>=2.1.1. The default is otherwise the manual "eager" implementation.

Instantiates one of the model classes of the library (with a masked language modeling head) from a configuration.

Note: Loading a model from its configuration file does not load the model weights. It only affects the model’s configuration. Use from_pretrained() to load the model weights.

Examples:

>>> from transformers import AutoConfig, AutoModelForMaskedLM

>>> # Download configuration from huggingface.co and cache.
>>> config = AutoConfig.from_pretrained("google-bert/bert-base-cased")
>>> model = AutoModelForMaskedLM.from_config(config)

from_pretrained

< >

( *model_args **kwargs )

Parameters

  • pretrained_model_name_or_path (str or os.PathLike) — Can be either:

    • A string, the model id of a pretrained model hosted inside a model repo on huggingface.co.
    • A path to a directory containing model weights saved using save_pretrained(), e.g., ./my_model_directory/.
    • A path or url to a tensorflow index checkpoint file (e.g, ./tf_model/model.ckpt.index). In this case, from_tf should be set to True and a configuration object should be provided as config argument. This loading path is slower than converting the TensorFlow checkpoint in a PyTorch model using the provided conversion scripts and loading the PyTorch model afterwards.
  • model_args (additional positional arguments, optional) — Will be passed along to the underlying model __init__() method.
  • config (PretrainedConfig, optional) — Configuration for the model to use instead of an automatically loaded configuration. Configuration can be automatically loaded when:

    • The model is a model provided by the library (loaded with the model id string of a pretrained model).
    • The model was saved using save_pretrained() and is reloaded by supplying the save directory.
    • The model is loaded by supplying a local directory as pretrained_model_name_or_path and a configuration JSON file named config.json is found in the directory.
  • state_dict (Dict[str, torch.Tensor], optional) — A state dictionary to use instead of a state dictionary loaded from saved weights file.

    This option can be used if you want to create a model from a pretrained configuration but load your own weights. In this case though, you should check if using save_pretrained() and from_pretrained() is not a simpler option.

  • cache_dir (str or os.PathLike, optional) — Path to a directory in which a downloaded pretrained model configuration should be cached if the standard cache should not be used.
  • from_tf (bool, optional, defaults to False) — Load the model weights from a TensorFlow checkpoint save file (see docstring of pretrained_model_name_or_path argument).
  • force_download (bool, optional, defaults to False) — Whether or not to force the (re-)download of the model weights and configuration files, overriding the cached versions if they exist.
  • resume_download (bool, optional, defaults to False) — Whether or not to delete incompletely received files. Will attempt to resume the download if such a file exists.
  • proxies (Dict[str, str], optional) — A dictionary of proxy servers to use by protocol or endpoint, e.g., {'http': 'foo.bar:3128', 'http://hostname': 'foo.bar:4012'}. The proxies are used on each request.
  • output_loading_info(bool, optional, defaults to False) — Whether ot not to also return a dictionary containing missing keys, unexpected keys and error messages.
  • local_files_only(bool, optional, defaults to False) — Whether or not to only look at local files (e.g., not try downloading the model).
  • revision (str, optional, defaults to "main") — The specific model version to use. It can be a branch name, a tag name, or a commit id, since we use a git-based system for storing models and other artifacts on huggingface.co, so revision can be any identifier allowed by git.
  • trust_remote_code (bool, optional, defaults to False) — Whether or not to allow for custom models defined on the Hub in their own modeling files. This option should only be set to True for repositories you trust and in which you have read the code, as it will execute code present on the Hub on your local machine.
  • code_revision (str, optional, defaults to "main") — The specific revision to use for the code on the Hub, if the code leaves in a different repository than the rest of the model. It can be a branch name, a tag name, or a commit id, since we use a git-based system for storing models and other artifacts on huggingface.co, so revision can be any identifier allowed by git.
  • kwargs (additional keyword arguments, optional) — Can be used to update the configuration object (after it being loaded) and initiate the model (e.g., output_attentions=True). Behaves differently depending on whether a config is provided or automatically loaded:

    • If a configuration is provided with config, **kwargs will be directly passed to the underlying model’s __init__ method (we assume all relevant updates to the configuration have already been done)
    • If a configuration is not provided, kwargs will be first passed to the configuration class initialization function (from_pretrained()). Each key of kwargs that corresponds to a configuration attribute will be used to override said attribute with the supplied kwargs value. Remaining keys that do not correspond to any configuration attribute will be passed to the underlying model’s __init__ function.

Instantiate one of the model classes of the library (with a masked language modeling head) from a pretrained model.

The model class to instantiate is selected based on the model_type property of the config object (either passed as an argument or loaded from pretrained_model_name_or_path if possible), or when it’s missing, by falling back to using pattern matching on pretrained_model_name_or_path:

  • albert — AlbertForMaskedLM (ALBERT model)
  • bart — BartForConditionalGeneration (BART model)
  • bert — BertForMaskedLM (BERT model)
  • big_bird — BigBirdForMaskedLM (BigBird model)
  • camembert — CamembertForMaskedLM (CamemBERT model)
  • convbert — ConvBertForMaskedLM (ConvBERT model)
  • data2vec-text — Data2VecTextForMaskedLM (Data2VecText model)
  • deberta — DebertaForMaskedLM (DeBERTa model)
  • deberta-v2 — DebertaV2ForMaskedLM (DeBERTa-v2 model)
  • distilbert — DistilBertForMaskedLM (DistilBERT model)
  • electra — ElectraForMaskedLM (ELECTRA model)
  • ernie — ErnieForMaskedLM (ERNIE model)
  • esm — EsmForMaskedLM (ESM model)
  • flaubert — FlaubertWithLMHeadModel (FlauBERT model)
  • fnet — FNetForMaskedLM (FNet model)
  • funnel — FunnelForMaskedLM (Funnel Transformer model)
  • ibert — IBertForMaskedLM (I-BERT model)
  • layoutlm — LayoutLMForMaskedLM (LayoutLM model)
  • longformer — LongformerForMaskedLM (Longformer model)
  • luke — LukeForMaskedLM (LUKE model)
  • mbart — MBartForConditionalGeneration (mBART model)
  • mega — MegaForMaskedLM (MEGA model)
  • megatron-bert — MegatronBertForMaskedLM (Megatron-BERT model)
  • mobilebert — MobileBertForMaskedLM (MobileBERT model)
  • mpnet — MPNetForMaskedLM (MPNet model)
  • mra — MraForMaskedLM (MRA model)
  • mvp — MvpForConditionalGeneration (MVP model)
  • nezha — NezhaForMaskedLM (Nezha model)
  • nystromformer — NystromformerForMaskedLM (Nyströmformer model)
  • perceiver — PerceiverForMaskedLM (Perceiver model)
  • qdqbert — QDQBertForMaskedLM (QDQBert model)
  • reformer — ReformerForMaskedLM (Reformer model)
  • rembert — RemBertForMaskedLM (RemBERT model)
  • roberta — RobertaForMaskedLM (RoBERTa model)
  • roberta-prelayernorm — RobertaPreLayerNormForMaskedLM (RoBERTa-PreLayerNorm model)
  • roc_bert — RoCBertForMaskedLM (RoCBert model)
  • roformer — RoFormerForMaskedLM (RoFormer model)
  • squeezebert — SqueezeBertForMaskedLM (SqueezeBERT model)
  • tapas — TapasForMaskedLM (TAPAS model)
  • wav2vec2 — Wav2Vec2ForMaskedLM (Wav2Vec2 model)
  • xlm — XLMWithLMHeadModel (XLM model)
  • xlm-roberta — XLMRobertaForMaskedLM (XLM-RoBERTa model)
  • xlm-roberta-xl — XLMRobertaXLForMaskedLM (XLM-RoBERTa-XL model)
  • xmod — XmodForMaskedLM (X-MOD model)
  • yoso — YosoForMaskedLM (YOSO model)

The model is set in evaluation mode by default using model.eval() (so for instance, dropout modules are deactivated). To train the model, you should first set it back in training mode with model.train()

Examples:

>>> from transformers import AutoConfig, AutoModelForMaskedLM

>>> # Download model and configuration from huggingface.co and cache.
>>> model = AutoModelForMaskedLM.from_pretrained("google-bert/bert-base-cased")

>>> # Update configuration during loading
>>> model = AutoModelForMaskedLM.from_pretrained("google-bert/bert-base-cased", output_attentions=True)
>>> model.config.output_attentions
True

>>> # Loading from a TF checkpoint file instead of a PyTorch model (slower)
>>> config = AutoConfig.from_pretrained("./tf_model/bert_tf_model_config.json")
>>> model = AutoModelForMaskedLM.from_pretrained(
...     "./tf_model/bert_tf_checkpoint.ckpt.index", from_tf=True, config=config
... )

TFAutoModelForMaskedLM

class transformers.TFAutoModelForMaskedLM

< >

( *args **kwargs )

This is a generic model class that will be instantiated as one of the model classes of the library (with a masked language modeling head) when created with the from_pretrained() class method or the from_config() class method.

This class cannot be instantiated directly using __init__() (throws an error).

from_config

< >

( **kwargs )

Parameters

  • config (PretrainedConfig) — The model class to instantiate is selected based on the configuration class:

    • AlbertConfig configuration class: TFAlbertForMaskedLM (ALBERT model)
    • BertConfig configuration class: TFBertForMaskedLM (BERT model)
    • CamembertConfig configuration class: TFCamembertForMaskedLM (CamemBERT model)
    • ConvBertConfig configuration class: TFConvBertForMaskedLM (ConvBERT model)
    • DebertaConfig configuration class: TFDebertaForMaskedLM (DeBERTa model)
    • DebertaV2Config configuration class: TFDebertaV2ForMaskedLM (DeBERTa-v2 model)
    • DistilBertConfig configuration class: TFDistilBertForMaskedLM (DistilBERT model)
    • ElectraConfig configuration class: TFElectraForMaskedLM (ELECTRA model)
    • EsmConfig configuration class: TFEsmForMaskedLM (ESM model)
    • FlaubertConfig configuration class: TFFlaubertWithLMHeadModel (FlauBERT model)
    • FunnelConfig configuration class: TFFunnelForMaskedLM (Funnel Transformer model)
    • LayoutLMConfig configuration class: TFLayoutLMForMaskedLM (LayoutLM model)
    • LongformerConfig configuration class: TFLongformerForMaskedLM (Longformer model)
    • MPNetConfig configuration class: TFMPNetForMaskedLM (MPNet model)
    • MobileBertConfig configuration class: TFMobileBertForMaskedLM (MobileBERT model)
    • RemBertConfig configuration class: TFRemBertForMaskedLM (RemBERT model)
    • RoFormerConfig configuration class: TFRoFormerForMaskedLM (RoFormer model)
    • RobertaConfig configuration class: TFRobertaForMaskedLM (RoBERTa model)
    • RobertaPreLayerNormConfig configuration class: TFRobertaPreLayerNormForMaskedLM (RoBERTa-PreLayerNorm model)
    • TapasConfig configuration class: TFTapasForMaskedLM (TAPAS model)
    • XLMConfig configuration class: TFXLMWithLMHeadModel (XLM model)
    • XLMRobertaConfig configuration class: TFXLMRobertaForMaskedLM (XLM-RoBERTa model)
  • attn_implementation (str, optional) — The attention implementation to use in the model (if relevant). Can be any of "eager" (manual implementation of the attention), "sdpa" (using F.scaled_dot_product_attention), or "flash_attention_2" (using Dao-AILab/flash-attention). By default, if available, SDPA will be used for torch>=2.1.1. The default is otherwise the manual "eager" implementation.

Instantiates one of the model classes of the library (with a masked language modeling head) from a configuration.

Note: Loading a model from its configuration file does not load the model weights. It only affects the model’s configuration. Use from_pretrained() to load the model weights.

Examples:

>>> from transformers import AutoConfig, TFAutoModelForMaskedLM

>>> # Download configuration from huggingface.co and cache.
>>> config = AutoConfig.from_pretrained("google-bert/bert-base-cased")
>>> model = TFAutoModelForMaskedLM.from_config(config)

from_pretrained

< >

( *model_args **kwargs )

Parameters

  • pretrained_model_name_or_path (str or os.PathLike) — Can be either:

    • A string, the model id of a pretrained model hosted inside a model repo on huggingface.co.
    • A path to a directory containing model weights saved using save_pretrained(), e.g., ./my_model_directory/.
    • A path or url to a PyTorch state_dict save file (e.g, ./pt_model/pytorch_model.bin). In this case, from_pt should be set to True and a configuration object should be provided as config argument. This loading path is slower than converting the PyTorch model in a TensorFlow model using the provided conversion scripts and loading the TensorFlow model afterwards.
  • model_args (additional positional arguments, optional) — Will be passed along to the underlying model __init__() method.
  • config (PretrainedConfig, optional) — Configuration for the model to use instead of an automatically loaded configuration. Configuration can be automatically loaded when:

    • The model is a model provided by the library (loaded with the model id string of a pretrained model).
    • The model was saved using save_pretrained() and is reloaded by supplying the save directory.
    • The model is loaded by supplying a local directory as pretrained_model_name_or_path and a configuration JSON file named config.json is found in the directory.
  • cache_dir (str or os.PathLike, optional) — Path to a directory in which a downloaded pretrained model configuration should be cached if the standard cache should not be used.
  • from_pt (bool, optional, defaults to False) — Load the model weights from a PyTorch checkpoint save file (see docstring of pretrained_model_name_or_path argument).
  • force_download (bool, optional, defaults to False) — Whether or not to force the (re-)download of the model weights and configuration files, overriding the cached versions if they exist.
  • resume_download (bool, optional, defaults to False) — Whether or not to delete incompletely received files. Will attempt to resume the download if such a file exists.
  • proxies (Dict[str, str], optional) — A dictionary of proxy servers to use by protocol or endpoint, e.g., {'http': 'foo.bar:3128', 'http://hostname': 'foo.bar:4012'}. The proxies are used on each request.
  • output_loading_info(bool, optional, defaults to False) — Whether ot not to also return a dictionary containing missing keys, unexpected keys and error messages.
  • local_files_only(bool, optional, defaults to False) — Whether or not to only look at local files (e.g., not try downloading the model).
  • revision (str, optional, defaults to "main") — The specific model version to use. It can be a branch name, a tag name, or a commit id, since we use a git-based system for storing models and other artifacts on huggingface.co, so revision can be any identifier allowed by git.
  • trust_remote_code (bool, optional, defaults to False) — Whether or not to allow for custom models defined on the Hub in their own modeling files. This option should only be set to True for repositories you trust and in which you have read the code, as it will execute code present on the Hub on your local machine.
  • code_revision (str, optional, defaults to "main") — The specific revision to use for the code on the Hub, if the code leaves in a different repository than the rest of the model. It can be a branch name, a tag name, or a commit id, since we use a git-based system for storing models and other artifacts on huggingface.co, so revision can be any identifier allowed by git.
  • kwargs (additional keyword arguments, optional) — Can be used to update the configuration object (after it being loaded) and initiate the model (e.g., output_attentions=True). Behaves differently depending on whether a config is provided or automatically loaded:

    • If a configuration is provided with config, **kwargs will be directly passed to the underlying model’s __init__ method (we assume all relevant updates to the configuration have already been done)
    • If a configuration is not provided, kwargs will be first passed to the configuration class initialization function (from_pretrained()). Each key of kwargs that corresponds to a configuration attribute will be used to override said attribute with the supplied kwargs value. Remaining keys that do not correspond to any configuration attribute will be passed to the underlying model’s __init__ function.

Instantiate one of the model classes of the library (with a masked language modeling head) from a pretrained model.

The model class to instantiate is selected based on the model_type property of the config object (either passed as an argument or loaded from pretrained_model_name_or_path if possible), or when it’s missing, by falling back to using pattern matching on pretrained_model_name_or_path:

  • albert — TFAlbertForMaskedLM (ALBERT model)
  • bert — TFBertForMaskedLM (BERT model)
  • camembert — TFCamembertForMaskedLM (CamemBERT model)
  • convbert — TFConvBertForMaskedLM (ConvBERT model)
  • deberta — TFDebertaForMaskedLM (DeBERTa model)
  • deberta-v2 — TFDebertaV2ForMaskedLM (DeBERTa-v2 model)
  • distilbert — TFDistilBertForMaskedLM (DistilBERT model)
  • electra — TFElectraForMaskedLM (ELECTRA model)
  • esm — TFEsmForMaskedLM (ESM model)
  • flaubert — TFFlaubertWithLMHeadModel (FlauBERT model)
  • funnel — TFFunnelForMaskedLM (Funnel Transformer model)
  • layoutlm — TFLayoutLMForMaskedLM (LayoutLM model)
  • longformer — TFLongformerForMaskedLM (Longformer model)
  • mobilebert — TFMobileBertForMaskedLM (MobileBERT model)
  • mpnet — TFMPNetForMaskedLM (MPNet model)
  • rembert — TFRemBertForMaskedLM (RemBERT model)
  • roberta — TFRobertaForMaskedLM (RoBERTa model)
  • roberta-prelayernorm — TFRobertaPreLayerNormForMaskedLM (RoBERTa-PreLayerNorm model)
  • roformer — TFRoFormerForMaskedLM (RoFormer model)
  • tapas — TFTapasForMaskedLM (TAPAS model)
  • xlm — TFXLMWithLMHeadModel (XLM model)
  • xlm-roberta — TFXLMRobertaForMaskedLM (XLM-RoBERTa model)

Examples:

>>> from transformers import AutoConfig, TFAutoModelForMaskedLM

>>> # Download model and configuration from huggingface.co and cache.
>>> model = TFAutoModelForMaskedLM.from_pretrained("google-bert/bert-base-cased")

>>> # Update configuration during loading
>>> model = TFAutoModelForMaskedLM.from_pretrained("google-bert/bert-base-cased", output_attentions=True)
>>> model.config.output_attentions
True

>>> # Loading from a PyTorch checkpoint file instead of a TensorFlow model (slower)
>>> config = AutoConfig.from_pretrained("./pt_model/bert_pt_model_config.json")
>>> model = TFAutoModelForMaskedLM.from_pretrained(
...     "./pt_model/bert_pytorch_model.bin", from_pt=True, config=config
... )

FlaxAutoModelForMaskedLM

class transformers.FlaxAutoModelForMaskedLM

< >

( *args **kwargs )

This is a generic model class that will be instantiated as one of the model classes of the library (with a masked language modeling head) when created with the from_pretrained() class method or the from_config() class method.

This class cannot be instantiated directly using __init__() (throws an error).

from_config

< >

( **kwargs )

Parameters

  • config (PretrainedConfig) — The model class to instantiate is selected based on the configuration class:

    • AlbertConfig configuration class: FlaxAlbertForMaskedLM (ALBERT model)
    • BartConfig configuration class: FlaxBartForConditionalGeneration (BART model)
    • BertConfig configuration class: FlaxBertForMaskedLM (BERT model)
    • BigBirdConfig configuration class: FlaxBigBirdForMaskedLM (BigBird model)
    • DistilBertConfig configuration class: FlaxDistilBertForMaskedLM (DistilBERT model)
    • ElectraConfig configuration class: FlaxElectraForMaskedLM (ELECTRA model)
    • MBartConfig configuration class: FlaxMBartForConditionalGeneration (mBART model)
    • RoFormerConfig configuration class: FlaxRoFormerForMaskedLM (RoFormer model)
    • RobertaConfig configuration class: FlaxRobertaForMaskedLM (RoBERTa model)
    • RobertaPreLayerNormConfig configuration class: FlaxRobertaPreLayerNormForMaskedLM (RoBERTa-PreLayerNorm model)
    • XLMRobertaConfig configuration class: FlaxXLMRobertaForMaskedLM (XLM-RoBERTa model)
  • attn_implementation (str, optional) — The attention implementation to use in the model (if relevant). Can be any of "eager" (manual implementation of the attention), "sdpa" (using F.scaled_dot_product_attention), or "flash_attention_2" (using Dao-AILab/flash-attention). By default, if available, SDPA will be used for torch>=2.1.1. The default is otherwise the manual "eager" implementation.

Instantiates one of the model classes of the library (with a masked language modeling head) from a configuration.

Note: Loading a model from its configuration file does not load the model weights. It only affects the model’s configuration. Use from_pretrained() to load the model weights.

Examples:

>>> from transformers import AutoConfig, FlaxAutoModelForMaskedLM

>>> # Download configuration from huggingface.co and cache.
>>> config = AutoConfig.from_pretrained("google-bert/bert-base-cased")
>>> model = FlaxAutoModelForMaskedLM.from_config(config)

from_pretrained

< >

( *model_args **kwargs )

Parameters

  • pretrained_model_name_or_path (str or os.PathLike) — Can be either:

    • A string, the model id of a pretrained model hosted inside a model repo on huggingface.co.
    • A path to a directory containing model weights saved using save_pretrained(), e.g., ./my_model_directory/.
    • A path or url to a PyTorch state_dict save file (e.g, ./pt_model/pytorch_model.bin). In this case, from_pt should be set to True and a configuration object should be provided as config argument. This loading path is slower than converting the PyTorch model in a TensorFlow model using the provided conversion scripts and loading the TensorFlow model afterwards.
  • model_args (additional positional arguments, optional) — Will be passed along to the underlying model __init__() method.
  • config (PretrainedConfig, optional) — Configuration for the model to use instead of an automatically loaded configuration. Configuration can be automatically loaded when:

    • The model is a model provided by the library (loaded with the model id string of a pretrained model).
    • The model was saved using save_pretrained() and is reloaded by supplying the save directory.
    • The model is loaded by supplying a local directory as pretrained_model_name_or_path and a configuration JSON file named config.json is found in the directory.
  • cache_dir (str or os.PathLike, optional) — Path to a directory in which a downloaded pretrained model configuration should be cached if the standard cache should not be used.
  • from_pt (bool, optional, defaults to False) — Load the model weights from a PyTorch checkpoint save file (see docstring of pretrained_model_name_or_path argument).
  • force_download (bool, optional, defaults to False) — Whether or not to force the (re-)download of the model weights and configuration files, overriding the cached versions if they exist.
  • resume_download (bool, optional, defaults to False) — Whether or not to delete incompletely received files. Will attempt to resume the download if such a file exists.
  • proxies (Dict[str, str], optional) — A dictionary of proxy servers to use by protocol or endpoint, e.g., {'http': 'foo.bar:3128', 'http://hostname': 'foo.bar:4012'}. The proxies are used on each request.
  • output_loading_info(bool, optional, defaults to False) — Whether ot not to also return a dictionary containing missing keys, unexpected keys and error messages.
  • local_files_only(bool, optional, defaults to False) — Whether or not to only look at local files (e.g., not try downloading the model).
  • revision (str, optional, defaults to "main") — The specific model version to use. It can be a branch name, a tag name, or a commit id, since we use a git-based system for storing models and other artifacts on huggingface.co, so revision can be any identifier allowed by git.
  • trust_remote_code (bool, optional, defaults to False) — Whether or not to allow for custom models defined on the Hub in their own modeling files. This option should only be set to True for repositories you trust and in which you have read the code, as it will execute code present on the Hub on your local machine.
  • code_revision (str, optional, defaults to "main") — The specific revision to use for the code on the Hub, if the code leaves in a different repository than the rest of the model. It can be a branch name, a tag name, or a commit id, since we use a git-based system for storing models and other artifacts on huggingface.co, so revision can be any identifier allowed by git.
  • kwargs (additional keyword arguments, optional) — Can be used to update the configuration object (after it being loaded) and initiate the model (e.g., output_attentions=True). Behaves differently depending on whether a config is provided or automatically loaded:

    • If a configuration is provided with config, **kwargs will be directly passed to the underlying model’s __init__ method (we assume all relevant updates to the configuration have already been done)
    • If a configuration is not provided, kwargs will be first passed to the configuration class initialization function (from_pretrained()). Each key of kwargs that corresponds to a configuration attribute will be used to override said attribute with the supplied kwargs value. Remaining keys that do not correspond to any configuration attribute will be passed to the underlying model’s __init__ function.

Instantiate one of the model classes of the library (with a masked language modeling head) from a pretrained model.

The model class to instantiate is selected based on the model_type property of the config object (either passed as an argument or loaded from pretrained_model_name_or_path if possible), or when it’s missing, by falling back to using pattern matching on pretrained_model_name_or_path:

  • albert — FlaxAlbertForMaskedLM (ALBERT model)
  • bart — FlaxBartForConditionalGeneration (BART model)
  • bert — FlaxBertForMaskedLM (BERT model)
  • big_bird — FlaxBigBirdForMaskedLM (BigBird model)
  • distilbert — FlaxDistilBertForMaskedLM (DistilBERT model)
  • electra — FlaxElectraForMaskedLM (ELECTRA model)
  • mbart — FlaxMBartForConditionalGeneration (mBART model)
  • roberta — FlaxRobertaForMaskedLM (RoBERTa model)
  • roberta-prelayernorm — FlaxRobertaPreLayerNormForMaskedLM (RoBERTa-PreLayerNorm model)
  • roformer — FlaxRoFormerForMaskedLM (RoFormer model)
  • xlm-roberta — FlaxXLMRobertaForMaskedLM (XLM-RoBERTa model)

Examples:

>>> from transformers import AutoConfig, FlaxAutoModelForMaskedLM

>>> # Download model and configuration from huggingface.co and cache.
>>> model = FlaxAutoModelForMaskedLM.from_pretrained("google-bert/bert-base-cased")

>>> # Update configuration during loading
>>> model = FlaxAutoModelForMaskedLM.from_pretrained("google-bert/bert-base-cased", output_attentions=True)
>>> model.config.output_attentions
True

>>> # Loading from a PyTorch checkpoint file instead of a TensorFlow model (slower)
>>> config = AutoConfig.from_pretrained("./pt_model/bert_pt_model_config.json")
>>> model = FlaxAutoModelForMaskedLM.from_pretrained(
...     "./pt_model/bert_pytorch_model.bin", from_pt=True, config=config
... )

AutoModelForMaskGeneration

class transformers.AutoModelForMaskGeneration

< >

( *args **kwargs )

TFAutoModelForMaskGeneration

class transformers.TFAutoModelForMaskGeneration

< >

( *args **kwargs )

AutoModelForSeq2SeqLM

class transformers.AutoModelForSeq2SeqLM

< >

( *args **kwargs )

This is a generic model class that will be instantiated as one of the model classes of the library (with a sequence-to-sequence language modeling head) when created with the from_pretrained() class method or the from_config() class method.

This class cannot be instantiated directly using __init__() (throws an error).

from_config

< >

( **kwargs )

Parameters

  • config (PretrainedConfig) — The model class to instantiate is selected based on the configuration class:

    • BartConfig configuration class: BartForConditionalGeneration (BART model)
    • BigBirdPegasusConfig configuration class: BigBirdPegasusForConditionalGeneration (BigBird-Pegasus model)
    • BlenderbotConfig configuration class: BlenderbotForConditionalGeneration (Blenderbot model)
    • BlenderbotSmallConfig configuration class: BlenderbotSmallForConditionalGeneration (BlenderbotSmall model)
    • EncoderDecoderConfig configuration class: EncoderDecoderModel (Encoder decoder model)
    • FSMTConfig configuration class: FSMTForConditionalGeneration (FairSeq Machine-Translation model)
    • GPTSanJapaneseConfig configuration class: GPTSanJapaneseForConditionalGeneration (GPTSAN-japanese model)
    • LEDConfig configuration class: LEDForConditionalGeneration (LED model)
    • LongT5Config configuration class: LongT5ForConditionalGeneration (LongT5 model)
    • M2M100Config configuration class: M2M100ForConditionalGeneration (M2M100 model)
    • MBartConfig configuration class: MBartForConditionalGeneration (mBART model)
    • MT5Config configuration class: MT5ForConditionalGeneration (MT5 model)
    • MarianConfig configuration class: MarianMTModel (Marian model)
    • MvpConfig configuration class: MvpForConditionalGeneration (MVP model)
    • NllbMoeConfig configuration class: NllbMoeForConditionalGeneration (NLLB-MOE model)
    • PLBartConfig configuration class: PLBartForConditionalGeneration (PLBart model)
    • PegasusConfig configuration class: PegasusForConditionalGeneration (Pegasus model)
    • PegasusXConfig configuration class: PegasusXForConditionalGeneration (PEGASUS-X model)
    • ProphetNetConfig configuration class: ProphetNetForConditionalGeneration (ProphetNet model)
    • SeamlessM4TConfig configuration class: SeamlessM4TForTextToText (SeamlessM4T model)
    • SeamlessM4Tv2Config configuration class: SeamlessM4Tv2ForTextToText (SeamlessM4Tv2 model)
    • SwitchTransformersConfig configuration class: SwitchTransformersForConditionalGeneration (SwitchTransformers model)
    • T5Config configuration class: T5ForConditionalGeneration (T5 model)
    • UMT5Config configuration class: UMT5ForConditionalGeneration (UMT5 model)
    • XLMProphetNetConfig configuration class: XLMProphetNetForConditionalGeneration (XLM-ProphetNet model)
  • attn_implementation (str, optional) — The attention implementation to use in the model (if relevant). Can be any of "eager" (manual implementation of the attention), "sdpa" (using F.scaled_dot_product_attention), or "flash_attention_2" (using Dao-AILab/flash-attention). By default, if available, SDPA will be used for torch>=2.1.1. The default is otherwise the manual "eager" implementation.

Instantiates one of the model classes of the library (with a sequence-to-sequence language modeling head) from a configuration.

Note: Loading a model from its configuration file does not load the model weights. It only affects the model’s configuration. Use from_pretrained() to load the model weights.

Examples:

>>> from transformers import AutoConfig, AutoModelForSeq2SeqLM

>>> # Download configuration from huggingface.co and cache.
>>> config = AutoConfig.from_pretrained("google-t5/t5-base")
>>> model = AutoModelForSeq2SeqLM.from_config(config)

from_pretrained

< >

( *model_args **kwargs )

Parameters

  • pretrained_model_name_or_path (str or os.PathLike) — Can be either:

    • A string, the model id of a pretrained model hosted inside a model repo on huggingface.co.
    • A path to a directory containing model weights saved using save_pretrained(), e.g., ./my_model_directory/.
    • A path or url to a tensorflow index checkpoint file (e.g, ./tf_model/model.ckpt.index). In this case, from_tf should be set to True and a configuration object should be provided as config argument. This loading path is slower than converting the TensorFlow checkpoint in a PyTorch model using the provided conversion scripts and loading the PyTorch model afterwards.
  • model_args (additional positional arguments, optional) — Will be passed along to the underlying model __init__() method.
  • config (PretrainedConfig, optional) — Configuration for the model to use instead of an automatically loaded configuration. Configuration can be automatically loaded when:

    • The model is a model provided by the library (loaded with the model id string of a pretrained model).
    • The model was saved using save_pretrained() and is reloaded by supplying the save directory.
    • The model is loaded by supplying a local directory as pretrained_model_name_or_path and a configuration JSON file named config.json is found in the directory.
  • state_dict (Dict[str, torch.Tensor], optional) — A state dictionary to use instead of a state dictionary loaded from saved weights file.

    This option can be used if you want to create a model from a pretrained configuration but load your own weights. In this case though, you should check if using save_pretrained() and from_pretrained() is not a simpler option.

  • cache_dir (str or os.PathLike, optional) — Path to a directory in which a downloaded pretrained model configuration should be cached if the standard cache should not be used.
  • from_tf (bool, optional, defaults to False) — Load the model weights from a TensorFlow checkpoint save file (see docstring of pretrained_model_name_or_path argument).
  • force_download (bool, optional, defaults to False) — Whether or not to force the (re-)download of the model weights and configuration files, overriding the cached versions if they exist.
  • resume_download (bool, optional, defaults to False) — Whether or not to delete incompletely received files. Will attempt to resume the download if such a file exists.
  • proxies (Dict[str, str], optional) — A dictionary of proxy servers to use by protocol or endpoint, e.g., {'http': 'foo.bar:3128', 'http://hostname': 'foo.bar:4012'}. The proxies are used on each request.
  • output_loading_info(bool, optional, defaults to False) — Whether ot not to also return a dictionary containing missing keys, unexpected keys and error messages.
  • local_files_only(bool, optional, defaults to False) — Whether or not to only look at local files (e.g., not try downloading the model).
  • revision (str, optional, defaults to "main") — The specific model version to use. It can be a branch name, a tag name, or a commit id, since we use a git-based system for storing models and other artifacts on huggingface.co, so revision can be any identifier allowed by git.
  • trust_remote_code (bool, optional, defaults to False) — Whether or not to allow for custom models defined on the Hub in their own modeling files. This option should only be set to True for repositories you trust and in which you have read the code, as it will execute code present on the Hub on your local machine.
  • code_revision (str, optional, defaults to "main") — The specific revision to use for the code on the Hub, if the code leaves in a different repository than the rest of the model. It can be a branch name, a tag name, or a commit id, since we use a git-based system for storing models and other artifacts on huggingface.co, so revision can be any identifier allowed by git.
  • kwargs (additional keyword arguments, optional) — Can be used to update the configuration object (after it being loaded) and initiate the model (e.g., output_attentions=True). Behaves differently depending on whether a config is provided or automatically loaded:

    • If a configuration is provided with config, **kwargs will be directly passed to the underlying model’s __init__ method (we assume all relevant updates to the configuration have already been done)
    • If a configuration is not provided, kwargs will be first passed to the configuration class initialization function (from_pretrained()). Each key of kwargs that corresponds to a configuration attribute will be used to override said attribute with the supplied kwargs value. Remaining keys that do not correspond to any configuration attribute will be passed to the underlying model’s __init__ function.

Instantiate one of the model classes of the library (with a sequence-to-sequence language modeling head) from a pretrained model.

The model class to instantiate is selected based on the model_type property of the config object (either passed as an argument or loaded from pretrained_model_name_or_path if possible), or when it’s missing, by falling back to using pattern matching on pretrained_model_name_or_path:

  • bart — BartForConditionalGeneration (BART model)
  • bigbird_pegasus — BigBirdPegasusForConditionalGeneration (BigBird-Pegasus model)
  • blenderbot — BlenderbotForConditionalGeneration (Blenderbot model)
  • blenderbot-small — BlenderbotSmallForConditionalGeneration (BlenderbotSmall model)
  • encoder-decoder — EncoderDecoderModel (Encoder decoder model)
  • fsmt — FSMTForConditionalGeneration (FairSeq Machine-Translation model)
  • gptsan-japanese — GPTSanJapaneseForConditionalGeneration (GPTSAN-japanese model)
  • led — LEDForConditionalGeneration (LED model)
  • longt5 — LongT5ForConditionalGeneration (LongT5 model)
  • m2m_100 — M2M100ForConditionalGeneration (M2M100 model)
  • marian — MarianMTModel (Marian model)
  • mbart — MBartForConditionalGeneration (mBART model)
  • mt5 — MT5ForConditionalGeneration (MT5 model)
  • mvp — MvpForConditionalGeneration (MVP model)
  • nllb-moe — NllbMoeForConditionalGeneration (NLLB-MOE model)
  • pegasus — PegasusForConditionalGeneration (Pegasus model)
  • pegasus_x — PegasusXForConditionalGeneration (PEGASUS-X model)
  • plbart — PLBartForConditionalGeneration (PLBart model)
  • prophetnet — ProphetNetForConditionalGeneration (ProphetNet model)
  • seamless_m4t — SeamlessM4TForTextToText (SeamlessM4T model)
  • seamless_m4t_v2 — SeamlessM4Tv2ForTextToText (SeamlessM4Tv2 model)
  • switch_transformers — SwitchTransformersForConditionalGeneration (SwitchTransformers model)
  • t5 — T5ForConditionalGeneration (T5 model)
  • umt5 — UMT5ForConditionalGeneration (UMT5 model)
  • xlm-prophetnet — XLMProphetNetForConditionalGeneration (XLM-ProphetNet model)

The model is set in evaluation mode by default using model.eval() (so for instance, dropout modules are deactivated). To train the model, you should first set it back in training mode with model.train()

Examples:

>>> from transformers import AutoConfig, AutoModelForSeq2SeqLM

>>> # Download model and configuration from huggingface.co and cache.
>>> model = AutoModelForSeq2SeqLM.from_pretrained("google-t5/t5-base")

>>> # Update configuration during loading
>>> model = AutoModelForSeq2SeqLM.from_pretrained("google-t5/t5-base", output_attentions=True)
>>> model.config.output_attentions
True

>>> # Loading from a TF checkpoint file instead of a PyTorch model (slower)
>>> config = AutoConfig.from_pretrained("./tf_model/t5_tf_model_config.json")
>>> model = AutoModelForSeq2SeqLM.from_pretrained(
...     "./tf_model/t5_tf_checkpoint.ckpt.index", from_tf=True, config=config
... )

TFAutoModelForSeq2SeqLM

class transformers.TFAutoModelForSeq2SeqLM

< >

( *args **kwargs )

This is a generic model class that will be instantiated as one of the model classes of the library (with a sequence-to-sequence language modeling head) when created with the from_pretrained() class method or the from_config() class method.

This class cannot be instantiated directly using __init__() (throws an error).

from_config

< >

( **kwargs )

Parameters

  • config (PretrainedConfig) — The model class to instantiate is selected based on the configuration class:

    • BartConfig configuration class: TFBartForConditionalGeneration (BART model)
    • BlenderbotConfig configuration class: TFBlenderbotForConditionalGeneration (Blenderbot model)
    • BlenderbotSmallConfig configuration class: TFBlenderbotSmallForConditionalGeneration (BlenderbotSmall model)
    • EncoderDecoderConfig configuration class: TFEncoderDecoderModel (Encoder decoder model)
    • LEDConfig configuration class: TFLEDForConditionalGeneration (LED model)
    • MBartConfig configuration class: TFMBartForConditionalGeneration (mBART model)
    • MT5Config configuration class: TFMT5ForConditionalGeneration (MT5 model)
    • MarianConfig configuration class: TFMarianMTModel (Marian model)
    • PegasusConfig configuration class: TFPegasusForConditionalGeneration (Pegasus model)
    • T5Config configuration class: TFT5ForConditionalGeneration (T5 model)
  • attn_implementation (str, optional) — The attention implementation to use in the model (if relevant). Can be any of "eager" (manual implementation of the attention), "sdpa" (using F.scaled_dot_product_attention), or "flash_attention_2" (using Dao-AILab/flash-attention). By default, if available, SDPA will be used for torch>=2.1.1. The default is otherwise the manual "eager" implementation.

Instantiates one of the model classes of the library (with a sequence-to-sequence language modeling head) from a configuration.

Note: Loading a model from its configuration file does not load the model weights. It only affects the model’s configuration. Use from_pretrained() to load the model weights.

Examples:

>>> from transformers import AutoConfig, TFAutoModelForSeq2SeqLM

>>> # Download configuration from huggingface.co and cache.
>>> config = AutoConfig.from_pretrained("google-t5/t5-base")
>>> model = TFAutoModelForSeq2SeqLM.from_config(config)

from_pretrained

< >

( *model_args **kwargs )

Parameters

  • pretrained_model_name_or_path (str or os.PathLike) — Can be either:

    • A string, the model id of a pretrained model hosted inside a model repo on huggingface.co.
    • A path to a directory containing model weights saved using save_pretrained(), e.g., ./my_model_directory/.
    • A path or url to a PyTorch state_dict save file (e.g, ./pt_model/pytorch_model.bin). In this case, from_pt should be set to True and a configuration object should be provided as config argument. This loading path is slower than converting the PyTorch model in a TensorFlow model using the provided conversion scripts and loading the TensorFlow model afterwards.
  • model_args (additional positional arguments, optional) — Will be passed along to the underlying model __init__() method.
  • config (PretrainedConfig, optional) — Configuration for the model to use instead of an automatically loaded configuration. Configuration can be automatically loaded when:

    • The model is a model provided by the library (loaded with the model id string of a pretrained model).
    • The model was saved using save_pretrained() and is reloaded by supplying the save directory.
    • The model is loaded by supplying a local directory as pretrained_model_name_or_path and a configuration JSON file named config.json is found in the directory.
  • cache_dir (str or os.PathLike, optional) — Path to a directory in which a downloaded pretrained model configuration should be cached if the standard cache should not be used.
  • from_pt (bool, optional, defaults to False) — Load the model weights from a PyTorch checkpoint save file (see docstring of pretrained_model_name_or_path argument).
  • force_download (bool, optional, defaults to False) — Whether or not to force the (re-)download of the model weights and configuration files, overriding the cached versions if they exist.
  • resume_download (bool, optional, defaults to False) — Whether or not to delete incompletely received files. Will attempt to resume the download if such a file exists.
  • proxies (Dict[str, str], optional) — A dictionary of proxy servers to use by protocol or endpoint, e.g., {'http': 'foo.bar:3128', 'http://hostname': 'foo.bar:4012'}. The proxies are used on each request.
  • output_loading_info(bool, optional, defaults to False) — Whether ot not to also return a dictionary containing missing keys, unexpected keys and error messages.
  • local_files_only(bool, optional, defaults to False) — Whether or not to only look at local files (e.g., not try downloading the model).
  • revision (str, optional, defaults to "main") — The specific model version to use. It can be a branch name, a tag name, or a commit id, since we use a git-based system for storing models and other artifacts on huggingface.co, so revision can be any identifier allowed by git.
  • trust_remote_code (bool, optional, defaults to False) — Whether or not to allow for custom models defined on the Hub in their own modeling files. This option should only be set to True for repositories you trust and in which you have read the code, as it will execute code present on the Hub on your local machine.
  • code_revision (str, optional, defaults to "main") — The specific revision to use for the code on the Hub, if the code leaves in a different repository than the rest of the model. It can be a branch name, a tag name, or a commit id, since we use a git-based system for storing models and other artifacts on huggingface.co, so revision can be any identifier allowed by git.
  • kwargs (additional keyword arguments, optional) — Can be used to update the configuration object (after it being loaded) and initiate the model (e.g., output_attentions=True). Behaves differently depending on whether a config is provided or automatically loaded:

    • If a configuration is provided with config, **kwargs will be directly passed to the underlying model’s __init__ method (we assume all relevant updates to the configuration have already been done)
    • If a configuration is not provided, kwargs will be first passed to the configuration class initialization function (from_pretrained()). Each key of kwargs that corresponds to a configuration attribute will be used to override said attribute with the supplied kwargs value. Remaining keys that do not correspond to any configuration attribute will be passed to the underlying model’s __init__ function.

Instantiate one of the model classes of the library (with a sequence-to-sequence language modeling head) from a pretrained model.

The model class to instantiate is selected based on the model_type property of the config object (either passed as an argument or loaded from pretrained_model_name_or_path if possible), or when it’s missing, by falling back to using pattern matching on pretrained_model_name_or_path:

  • bart — TFBartForConditionalGeneration (BART model)
  • blenderbot — TFBlenderbotForConditionalGeneration (Blenderbot model)
  • blenderbot-small — TFBlenderbotSmallForConditionalGeneration (BlenderbotSmall model)
  • encoder-decoder — TFEncoderDecoderModel (Encoder decoder model)
  • led — TFLEDForConditionalGeneration (LED model)
  • marian — TFMarianMTModel (Marian model)
  • mbart — TFMBartForConditionalGeneration (mBART model)
  • mt5 — TFMT5ForConditionalGeneration (MT5 model)
  • pegasus — TFPegasusForConditionalGeneration (Pegasus model)
  • t5 — TFT5ForConditionalGeneration (T5 model)

Examples:

>>> from transformers import AutoConfig, TFAutoModelForSeq2SeqLM

>>> # Download model and configuration from huggingface.co and cache.
>>> model = TFAutoModelForSeq2SeqLM.from_pretrained("google-t5/t5-base")

>>> # Update configuration during loading
>>> model = TFAutoModelForSeq2SeqLM.from_pretrained("google-t5/t5-base", output_attentions=True)
>>> model.config.output_attentions
True

>>> # Loading from a PyTorch checkpoint file instead of a TensorFlow model (slower)
>>> config = AutoConfig.from_pretrained("./pt_model/t5_pt_model_config.json")
>>> model = TFAutoModelForSeq2SeqLM.from_pretrained(
...     "./pt_model/t5_pytorch_model.bin", from_pt=True, config=config
... )

FlaxAutoModelForSeq2SeqLM

class transformers.FlaxAutoModelForSeq2SeqLM

< >

( *args **kwargs )

This is a generic model class that will be instantiated as one of the model classes of the library (with a sequence-to-sequence language modeling head) when created with the from_pretrained() class method or the from_config() class method.

This class cannot be instantiated directly using __init__() (throws an error).

from_config

< >

( **kwargs )

Parameters

  • config (PretrainedConfig) — The model class to instantiate is selected based on the configuration class:

    • BartConfig configuration class: FlaxBartForConditionalGeneration (BART model)
    • BlenderbotConfig configuration class: FlaxBlenderbotForConditionalGeneration (Blenderbot model)
    • BlenderbotSmallConfig configuration class: FlaxBlenderbotSmallForConditionalGeneration (BlenderbotSmall model)
    • EncoderDecoderConfig configuration class: FlaxEncoderDecoderModel (Encoder decoder model)
    • LongT5Config configuration class: FlaxLongT5ForConditionalGeneration (LongT5 model)
    • MBartConfig configuration class: FlaxMBartForConditionalGeneration (mBART model)
    • MT5Config configuration class: FlaxMT5ForConditionalGeneration (MT5 model)
    • MarianConfig configuration class: FlaxMarianMTModel (Marian model)
    • PegasusConfig configuration class: FlaxPegasusForConditionalGeneration (Pegasus model)
    • T5Config configuration class: FlaxT5ForConditionalGeneration (T5 model)
  • attn_implementation (str, optional) — The attention implementation to use in the model (if relevant). Can be any of "eager" (manual implementation of the attention), "sdpa" (using F.scaled_dot_product_attention), or "flash_attention_2" (using Dao-AILab/flash-attention). By default, if available, SDPA will be used for torch>=2.1.1. The default is otherwise the manual "eager" implementation.

Instantiates one of the model classes of the library (with a sequence-to-sequence language modeling head) from a configuration.

Note: Loading a model from its configuration file does not load the model weights. It only affects the model’s configuration. Use from_pretrained() to load the model weights.

Examples:

>>> from transformers import AutoConfig, FlaxAutoModelForSeq2SeqLM

>>> # Download configuration from huggingface.co and cache.
>>> config = AutoConfig.from_pretrained("google-t5/t5-base")
>>> model = FlaxAutoModelForSeq2SeqLM.from_config(config)

from_pretrained

< >

( *model_args **kwargs )

Parameters

  • pretrained_model_name_or_path (str or os.PathLike) — Can be either:

    • A string, the model id of a pretrained model hosted inside a model repo on huggingface.co.
    • A path to a directory containing model weights saved using save_pretrained(), e.g., ./my_model_directory/.
    • A path or url to a PyTorch state_dict save file (e.g, ./pt_model/pytorch_model.bin). In this case, from_pt should be set to True and a configuration object should be provided as config argument. This loading path is slower than converting the PyTorch model in a TensorFlow model using the provided conversion scripts and loading the TensorFlow model afterwards.
  • model_args (additional positional arguments, optional) — Will be passed along to the underlying model __init__() method.
  • config (PretrainedConfig, optional) — Configuration for the model to use instead of an automatically loaded configuration. Configuration can be automatically loaded when:

    • The model is a model provided by the library (loaded with the model id string of a pretrained model).
    • The model was saved using save_pretrained() and is reloaded by supplying the save directory.
    • The model is loaded by supplying a local directory as pretrained_model_name_or_path and a configuration JSON file named config.json is found in the directory.
  • cache_dir (str or os.PathLike, optional) — Path to a directory in which a downloaded pretrained model configuration should be cached if the standard cache should not be used.
  • from_pt (bool, optional, defaults to False) — Load the model weights from a PyTorch checkpoint save file (see docstring of pretrained_model_name_or_path argument).
  • force_download (bool, optional, defaults to False) — Whether or not to force the (re-)download of the model weights and configuration files, overriding the cached versions if they exist.
  • resume_download (bool, optional, defaults to False) — Whether or not to delete incompletely received files. Will attempt to resume the download if such a file exists.
  • proxies (Dict[str, str], optional) — A dictionary of proxy servers to use by protocol or endpoint, e.g., {'http': 'foo.bar:3128', 'http://hostname': 'foo.bar:4012'}. The proxies are used on each request.
  • output_loading_info(bool, optional, defaults to False) — Whether ot not to also return a dictionary containing missing keys, unexpected keys and error messages.
  • local_files_only(bool, optional, defaults to False) — Whether or not to only look at local files (e.g., not try downloading the model).
  • revision (str, optional, defaults to "main") — The specific model version to use. It can be a branch name, a tag name, or a commit id, since we use a git-based system for storing models and other artifacts on huggingface.co, so revision can be any identifier allowed by git.
  • trust_remote_code (bool, optional, defaults to False) — Whether or not to allow for custom models defined on the Hub in their own modeling files. This option should only be set to True for repositories you trust and in which you have read the code, as it will execute code present on the Hub on your local machine.
  • code_revision (str, optional, defaults to "main") — The specific revision to use for the code on the Hub, if the code leaves in a different repository than the rest of the model. It can be a branch name, a tag name, or a commit id, since we use a git-based system for storing models and other artifacts on huggingface.co, so revision can be any identifier allowed by git.
  • kwargs (additional keyword arguments, optional) — Can be used to update the configuration object (after it being loaded) and initiate the model (e.g., output_attentions=True). Behaves differently depending on whether a config is provided or automatically loaded:

    • If a configuration is provided with config, **kwargs will be directly passed to the underlying model’s __init__ method (we assume all relevant updates to the configuration have already been done)
    • If a configuration is not provided, kwargs will be first passed to the configuration class initialization function (from_pretrained()). Each key of kwargs that corresponds to a configuration attribute will be used to override said attribute with the supplied kwargs value. Remaining keys that do not correspond to any configuration attribute will be passed to the underlying model’s __init__ function.

Instantiate one of the model classes of the library (with a sequence-to-sequence language modeling head) from a pretrained model.

The model class to instantiate is selected based on the model_type property of the config object (either passed as an argument or loaded from pretrained_model_name_or_path if possible), or when it’s missing, by falling back to using pattern matching on pretrained_model_name_or_path:

  • bart — FlaxBartForConditionalGeneration (BART model)
  • blenderbot — FlaxBlenderbotForConditionalGeneration (Blenderbot model)
  • blenderbot-small — FlaxBlenderbotSmallForConditionalGeneration (BlenderbotSmall model)
  • encoder-decoder — FlaxEncoderDecoderModel (Encoder decoder model)
  • longt5 — FlaxLongT5ForConditionalGeneration (LongT5 model)
  • marian — FlaxMarianMTModel (Marian model)
  • mbart — FlaxMBartForConditionalGeneration (mBART model)
  • mt5 — FlaxMT5ForConditionalGeneration (MT5 model)
  • pegasus — FlaxPegasusForConditionalGeneration (Pegasus model)
  • t5 — FlaxT5ForConditionalGeneration (T5 model)

Examples:

>>> from transformers import AutoConfig, FlaxAutoModelForSeq2SeqLM

>>> # Download model and configuration from huggingface.co and cache.
>>> model = FlaxAutoModelForSeq2SeqLM.from_pretrained("google-t5/t5-base")

>>> # Update configuration during loading
>>> model = FlaxAutoModelForSeq2SeqLM.from_pretrained("google-t5/t5-base", output_attentions=True)
>>> model.config.output_attentions
True

>>> # Loading from a PyTorch checkpoint file instead of a TensorFlow model (slower)
>>> config = AutoConfig.from_pretrained("./pt_model/t5_pt_model_config.json")
>>> model = FlaxAutoModelForSeq2SeqLM.from_pretrained(
...     "./pt_model/t5_pytorch_model.bin", from_pt=True, config=config
... )

AutoModelForSequenceClassification

class transformers.AutoModelForSequenceClassification

< >

( *args **kwargs )

This is a generic model class that will be instantiated as one of the model classes of the library (with a sequence classification head) when created with the from_pretrained() class method or the from_config() class method.

This class cannot be instantiated directly using __init__() (throws an error).

from_config

< >

( **kwargs )

Parameters

  • config (PretrainedConfig) — The model class to instantiate is selected based on the configuration class:

    • AlbertConfig configuration class: AlbertForSequenceClassification (ALBERT model)
    • BartConfig configuration class: BartForSequenceClassification (BART model)
    • BertConfig configuration class: BertForSequenceClassification (BERT model)
    • BigBirdConfig configuration class: BigBirdForSequenceClassification (BigBird model)
    • BigBirdPegasusConfig configuration class: BigBirdPegasusForSequenceClassification (BigBird-Pegasus model)
    • BioGptConfig configuration class: BioGptForSequenceClassification (BioGpt model)
    • BloomConfig configuration class: BloomForSequenceClassification (BLOOM model)
    • CTRLConfig configuration class: CTRLForSequenceClassification (CTRL model)
    • CamembertConfig configuration class: CamembertForSequenceClassification (CamemBERT model)
    • CanineConfig configuration class: CanineForSequenceClassification (CANINE model)
    • ConvBertConfig configuration class: ConvBertForSequenceClassification (ConvBERT model)
    • Data2VecTextConfig configuration class: Data2VecTextForSequenceClassification (Data2VecText model)
    • DebertaConfig configuration class: DebertaForSequenceClassification (DeBERTa model)
    • DebertaV2Config configuration class: DebertaV2ForSequenceClassification (DeBERTa-v2 model)
    • DistilBertConfig configuration class: DistilBertForSequenceClassification (DistilBERT model)
    • ElectraConfig configuration class: ElectraForSequenceClassification (ELECTRA model)
    • ErnieConfig configuration class: ErnieForSequenceClassification (ERNIE model)
    • ErnieMConfig configuration class: ErnieMForSequenceClassification (ErnieM model)
    • EsmConfig configuration class: EsmForSequenceClassification (ESM model)
    • FNetConfig configuration class: FNetForSequenceClassification (FNet model)
    • FalconConfig configuration class: FalconForSequenceClassification (Falcon model)
    • FlaubertConfig configuration class: FlaubertForSequenceClassification (FlauBERT model)
    • FunnelConfig configuration class: FunnelForSequenceClassification (Funnel Transformer model)
    • GPT2Config configuration class: GPT2ForSequenceClassification (OpenAI GPT-2 model)
    • GPTBigCodeConfig configuration class: GPTBigCodeForSequenceClassification (GPTBigCode model)
    • GPTJConfig configuration class: GPTJForSequenceClassification (GPT-J model)
    • GPTNeoConfig configuration class: GPTNeoForSequenceClassification (GPT Neo model)
    • GPTNeoXConfig configuration class: GPTNeoXForSequenceClassification (GPT NeoX model)
    • GemmaConfig configuration class: GemmaForSequenceClassification (Gemma model)
    • IBertConfig configuration class: IBertForSequenceClassification (I-BERT model)
    • JambaConfig configuration class: JambaForSequenceClassification (Jamba model)
    • LEDConfig configuration class: LEDForSequenceClassification (LED model)
    • LayoutLMConfig configuration class: LayoutLMForSequenceClassification (LayoutLM model)
    • LayoutLMv2Config configuration class: LayoutLMv2ForSequenceClassification (LayoutLMv2 model)
    • LayoutLMv3Config configuration class: LayoutLMv3ForSequenceClassification (LayoutLMv3 model)
    • LiltConfig configuration class: LiltForSequenceClassification (LiLT model)
    • LlamaConfig configuration class: LlamaForSequenceClassification (LLaMA model)
    • LongformerConfig configuration class: LongformerForSequenceClassification (Longformer model)
    • LukeConfig configuration class: LukeForSequenceClassification (LUKE model)
    • MBartConfig configuration class: MBartForSequenceClassification (mBART model)
    • MPNetConfig configuration class: MPNetForSequenceClassification (MPNet model)
    • MT5Config configuration class: MT5ForSequenceClassification (MT5 model)
    • MarkupLMConfig configuration class: MarkupLMForSequenceClassification (MarkupLM model)
    • MegaConfig configuration class: MegaForSequenceClassification (MEGA model)
    • MegatronBertConfig configuration class: MegatronBertForSequenceClassification (Megatron-BERT model)
    • MistralConfig configuration class: MistralForSequenceClassification (Mistral model)
    • MixtralConfig configuration class: MixtralForSequenceClassification (Mixtral model)
    • MobileBertConfig configuration class: MobileBertForSequenceClassification (MobileBERT model)
    • MptConfig configuration class: MptForSequenceClassification (MPT model)
    • MraConfig configuration class: MraForSequenceClassification (MRA model)
    • MvpConfig configuration class: MvpForSequenceClassification (MVP model)
    • NezhaConfig configuration class: NezhaForSequenceClassification (Nezha model)
    • NystromformerConfig configuration class: NystromformerForSequenceClassification (Nyströmformer model)
    • OPTConfig configuration class: OPTForSequenceClassification (OPT model)
    • OpenAIGPTConfig configuration class: OpenAIGPTForSequenceClassification (OpenAI GPT model)
    • OpenLlamaConfig configuration class: OpenLlamaForSequenceClassification (OpenLlama model)
    • PLBartConfig configuration class: PLBartForSequenceClassification (PLBart model)
    • PerceiverConfig configuration class: PerceiverForSequenceClassification (Perceiver model)
    • PersimmonConfig configuration class: PersimmonForSequenceClassification (Persimmon model)
    • PhiConfig configuration class: PhiForSequenceClassification (Phi model)
    • QDQBertConfig configuration class: QDQBertForSequenceClassification (QDQBert model)
    • Qwen2Config configuration class: Qwen2ForSequenceClassification (Qwen2 model)
    • Qwen2MoeConfig configuration class: Qwen2MoeForSequenceClassification (Qwen2MoE model)
    • ReformerConfig configuration class: ReformerForSequenceClassification (Reformer model)
    • RemBertConfig configuration class: RemBertForSequenceClassification (RemBERT model)
    • RoCBertConfig configuration class: RoCBertForSequenceClassification (RoCBert model)
    • RoFormerConfig configuration class: RoFormerForSequenceClassification (RoFormer model)
    • RobertaConfig configuration class: RobertaForSequenceClassification (RoBERTa model)
    • RobertaPreLayerNormConfig configuration class: RobertaPreLayerNormForSequenceClassification (RoBERTa-PreLayerNorm model)
    • SqueezeBertConfig configuration class: SqueezeBertForSequenceClassification (SqueezeBERT model)
    • StableLmConfig configuration class: StableLmForSequenceClassification (StableLm model)
    • Starcoder2Config configuration class: Starcoder2ForSequenceClassification (Starcoder2 model)
    • T5Config configuration class: T5ForSequenceClassification (T5 model)
    • TapasConfig configuration class: TapasForSequenceClassification (TAPAS model)
    • TransfoXLConfig configuration class: TransfoXLForSequenceClassification (Transformer-XL model)
    • UMT5Config configuration class: UMT5ForSequenceClassification (UMT5 model)
    • XLMConfig configuration class: XLMForSequenceClassification (XLM model)
    • XLMRobertaConfig configuration class: XLMRobertaForSequenceClassification (XLM-RoBERTa model)
    • XLMRobertaXLConfig configuration class: XLMRobertaXLForSequenceClassification (XLM-RoBERTa-XL model)
    • XLNetConfig configuration class: XLNetForSequenceClassification (XLNet model)
    • XmodConfig configuration class: XmodForSequenceClassification (X-MOD model)
    • YosoConfig configuration class: YosoForSequenceClassification (YOSO model)
  • attn_implementation (str, optional) — The attention implementation to use in the model (if relevant). Can be any of "eager" (manual implementation of the attention), "sdpa" (using F.scaled_dot_product_attention), or "flash_attention_2" (using Dao-AILab/flash-attention). By default, if available, SDPA will be used for torch>=2.1.1. The default is otherwise the manual "eager" implementation.

Instantiates one of the model classes of the library (with a sequence classification head) from a configuration.

Note: Loading a model from its configuration file does not load the model weights. It only affects the model’s configuration. Use from_pretrained() to load the model weights.

Examples:

>>> from transformers import AutoConfig, AutoModelForSequenceClassification

>>> # Download configuration from huggingface.co and cache.
>>> config = AutoConfig.from_pretrained("google-bert/bert-base-cased")
>>> model = AutoModelForSequenceClassification.from_config(config)

from_pretrained

< >

( *model_args **kwargs )

Parameters

  • pretrained_model_name_or_path (str or os.PathLike) — Can be either:

    • A string, the model id of a pretrained model hosted inside a model repo on huggingface.co.
    • A path to a directory containing model weights saved using save_pretrained(), e.g., ./my_model_directory/.
    • A path or url to a tensorflow index checkpoint file (e.g, ./tf_model/model.ckpt.index). In this case, from_tf should be set to True and a configuration object should be provided as config argument. This loading path is slower than converting the TensorFlow checkpoint in a PyTorch model using the provided conversion scripts and loading the PyTorch model afterwards.
  • model_args (additional positional arguments, optional) — Will be passed along to the underlying model __init__() method.
  • config (PretrainedConfig, optional) — Configuration for the model to use instead of an automatically loaded configuration. Configuration can be automatically loaded when:

    • The model is a model provided by the library (loaded with the model id string of a pretrained model).
    • The model was saved using save_pretrained() and is reloaded by supplying the save directory.
    • The model is loaded by supplying a local directory as pretrained_model_name_or_path and a configuration JSON file named config.json is found in the directory.
  • state_dict (Dict[str, torch.Tensor], optional) — A state dictionary to use instead of a state dictionary loaded from saved weights file.

    This option can be used if you want to create a model from a pretrained configuration but load your own weights. In this case though, you should check if using save_pretrained() and from_pretrained() is not a simpler option.

  • cache_dir (str or os.PathLike, optional) — Path to a directory in which a downloaded pretrained model configuration should be cached if the standard cache should not be used.
  • from_tf (bool, optional, defaults to False) — Load the model weights from a TensorFlow checkpoint save file (see docstring of pretrained_model_name_or_path argument).
  • force_download (bool, optional, defaults to False) — Whether or not to force the (re-)download of the model weights and configuration files, overriding the cached versions if they exist.
  • resume_download (bool, optional, defaults to False) — Whether or not to delete incompletely received files. Will attempt to resume the download if such a file exists.
  • proxies (Dict[str, str], optional) — A dictionary of proxy servers to use by protocol or endpoint, e.g., {'http': 'foo.bar:3128', 'http://hostname': 'foo.bar:4012'}. The proxies are used on each request.
  • output_loading_info(bool, optional, defaults to False) — Whether ot not to also return a dictionary containing missing keys, unexpected keys and error messages.
  • local_files_only(bool, optional, defaults to False) — Whether or not to only look at local files (e.g., not try downloading the model).
  • revision (str, optional, defaults to "main") — The specific model version to use. It can be a branch name, a tag name, or a commit id, since we use a git-based system for storing models and other artifacts on huggingface.co, so revision can be any identifier allowed by git.
  • trust_remote_code (bool, optional, defaults to False) — Whether or not to allow for custom models defined on the Hub in their own modeling files. This option should only be set to True for repositories you trust and in which you have read the code, as it will execute code present on the Hub on your local machine.
  • code_revision (str, optional, defaults to "main") — The specific revision to use for the code on the Hub, if the code leaves in a different repository than the rest of the model. It can be a branch name, a tag name, or a commit id, since we use a git-based system for storing models and other artifacts on huggingface.co, so revision can be any identifier allowed by git.
  • kwargs (additional keyword arguments, optional) — Can be used to update the configuration object (after it being loaded) and initiate the model (e.g., output_attentions=True). Behaves differently depending on whether a config is provided or automatically loaded:

    • If a configuration is provided with config, **kwargs will be directly passed to the underlying model’s __init__ method (we assume all relevant updates to the configuration have already been done)
    • If a configuration is not provided, kwargs will be first passed to the configuration class initialization function (from_pretrained()). Each key of kwargs that corresponds to a configuration attribute will be used to override said attribute with the supplied kwargs value. Remaining keys that do not correspond to any configuration attribute will be passed to the underlying model’s __init__ function.

Instantiate one of the model classes of the library (with a sequence classification head) from a pretrained model.

The model class to instantiate is selected based on the model_type property of the config object (either passed as an argument or loaded from pretrained_model_name_or_path if possible), or when it’s missing, by falling back to using pattern matching on pretrained_model_name_or_path:

  • albert — AlbertForSequenceClassification (ALBERT model)
  • bart — BartForSequenceClassification (BART model)
  • bert — BertForSequenceClassification (BERT model)
  • big_bird — BigBirdForSequenceClassification (BigBird model)
  • bigbird_pegasus — BigBirdPegasusForSequenceClassification (BigBird-Pegasus model)
  • biogpt — BioGptForSequenceClassification (BioGpt model)
  • bloom — BloomForSequenceClassification (BLOOM model)
  • camembert — CamembertForSequenceClassification (CamemBERT model)
  • canine — CanineForSequenceClassification (CANINE model)
  • code_llama — LlamaForSequenceClassification (CodeLlama model)
  • convbert — ConvBertForSequenceClassification (ConvBERT model)
  • ctrl — CTRLForSequenceClassification (CTRL model)
  • data2vec-text — Data2VecTextForSequenceClassification (Data2VecText model)
  • deberta — DebertaForSequenceClassification (DeBERTa model)
  • deberta-v2 — DebertaV2ForSequenceClassification (DeBERTa-v2 model)
  • distilbert — DistilBertForSequenceClassification (DistilBERT model)
  • electra — ElectraForSequenceClassification (ELECTRA model)
  • ernie — ErnieForSequenceClassification (ERNIE model)
  • ernie_m — ErnieMForSequenceClassification (ErnieM model)
  • esm — EsmForSequenceClassification (ESM model)
  • falcon — FalconForSequenceClassification (Falcon model)
  • flaubert — FlaubertForSequenceClassification (FlauBERT model)
  • fnet — FNetForSequenceClassification (FNet model)
  • funnel — FunnelForSequenceClassification (Funnel Transformer model)
  • gemma — GemmaForSequenceClassification (Gemma model)
  • gpt-sw3 — GPT2ForSequenceClassification (GPT-Sw3 model)
  • gpt2 — GPT2ForSequenceClassification (OpenAI GPT-2 model)
  • gpt_bigcode — GPTBigCodeForSequenceClassification (GPTBigCode model)
  • gpt_neo — GPTNeoForSequenceClassification (GPT Neo model)
  • gpt_neox — GPTNeoXForSequenceClassification (GPT NeoX model)
  • gptj — GPTJForSequenceClassification (GPT-J model)
  • ibert — IBertForSequenceClassification (I-BERT model)
  • jamba — JambaForSequenceClassification (Jamba model)
  • layoutlm — LayoutLMForSequenceClassification (LayoutLM model)
  • layoutlmv2 — LayoutLMv2ForSequenceClassification (LayoutLMv2 model)
  • layoutlmv3 — LayoutLMv3ForSequenceClassification (LayoutLMv3 model)
  • led — LEDForSequenceClassification (LED model)
  • lilt — LiltForSequenceClassification (LiLT model)
  • llama — LlamaForSequenceClassification (LLaMA model)
  • longformer — LongformerForSequenceClassification (Longformer model)
  • luke — LukeForSequenceClassification (LUKE model)
  • markuplm — MarkupLMForSequenceClassification (MarkupLM model)
  • mbart — MBartForSequenceClassification (mBART model)
  • mega — MegaForSequenceClassification (MEGA model)
  • megatron-bert — MegatronBertForSequenceClassification (Megatron-BERT model)
  • mistral — MistralForSequenceClassification (Mistral model)
  • mixtral — MixtralForSequenceClassification (Mixtral model)
  • mobilebert — MobileBertForSequenceClassification (MobileBERT model)
  • mpnet — MPNetForSequenceClassification (MPNet model)
  • mpt — MptForSequenceClassification (MPT model)
  • mra — MraForSequenceClassification (MRA model)
  • mt5 — MT5ForSequenceClassification (MT5 model)
  • mvp — MvpForSequenceClassification (MVP model)
  • nezha — NezhaForSequenceClassification (Nezha model)
  • nystromformer — NystromformerForSequenceClassification (Nyströmformer model)
  • open-llama — OpenLlamaForSequenceClassification (OpenLlama model)
  • openai-gpt — OpenAIGPTForSequenceClassification (OpenAI GPT model)
  • opt — OPTForSequenceClassification (OPT model)
  • perceiver — PerceiverForSequenceClassification (Perceiver model)
  • persimmon — PersimmonForSequenceClassification (Persimmon model)
  • phi — PhiForSequenceClassification (Phi model)
  • plbart — PLBartForSequenceClassification (PLBart model)
  • qdqbert — QDQBertForSequenceClassification (QDQBert model)
  • qwen2 — Qwen2ForSequenceClassification (Qwen2 model)
  • qwen2_moe — Qwen2MoeForSequenceClassification (Qwen2MoE model)
  • reformer — ReformerForSequenceClassification (Reformer model)
  • rembert — RemBertForSequenceClassification (RemBERT model)
  • roberta — RobertaForSequenceClassification (RoBERTa model)
  • roberta-prelayernorm — RobertaPreLayerNormForSequenceClassification (RoBERTa-PreLayerNorm model)
  • roc_bert — RoCBertForSequenceClassification (RoCBert model)
  • roformer — RoFormerForSequenceClassification (RoFormer model)
  • squeezebert — SqueezeBertForSequenceClassification (SqueezeBERT model)
  • stablelm — StableLmForSequenceClassification (StableLm model)
  • starcoder2 — Starcoder2ForSequenceClassification (Starcoder2 model)
  • t5 — T5ForSequenceClassification (T5 model)
  • tapas — TapasForSequenceClassification (TAPAS model)
  • transfo-xl — TransfoXLForSequenceClassification (Transformer-XL model)
  • umt5 — UMT5ForSequenceClassification (UMT5 model)
  • xlm — XLMForSequenceClassification (XLM model)
  • xlm-roberta — XLMRobertaForSequenceClassification (XLM-RoBERTa model)
  • xlm-roberta-xl — XLMRobertaXLForSequenceClassification (XLM-RoBERTa-XL model)
  • xlnet — XLNetForSequenceClassification (XLNet model)
  • xmod — XmodForSequenceClassification (X-MOD model)
  • yoso — YosoForSequenceClassification (YOSO model)

The model is set in evaluation mode by default using model.eval() (so for instance, dropout modules are deactivated). To train the model, you should first set it back in training mode with model.train()

Examples:

>>> from transformers import AutoConfig, AutoModelForSequenceClassification

>>> # Download model and configuration from huggingface.co and cache.
>>> model = AutoModelForSequenceClassification.from_pretrained("google-bert/bert-base-cased")

>>> # Update configuration during loading
>>> model = AutoModelForSequenceClassification.from_pretrained("google-bert/bert-base-cased", output_attentions=True)
>>> model.config.output_attentions
True

>>> # Loading from a TF checkpoint file instead of a PyTorch model (slower)
>>> config = AutoConfig.from_pretrained("./tf_model/bert_tf_model_config.json")
>>> model = AutoModelForSequenceClassification.from_pretrained(
...     "./tf_model/bert_tf_checkpoint.ckpt.index", from_tf=True, config=config
... )

TFAutoModelForSequenceClassification

class transformers.TFAutoModelForSequenceClassification

< >

( *args **kwargs )

This is a generic model class that will be instantiated as one of the model classes of the library (with a sequence classification head) when created with the from_pretrained() class method or the from_config() class method.

This class cannot be instantiated directly using __init__() (throws an error).

from_config

< >

( **kwargs )

Parameters

  • config (PretrainedConfig) — The model class to instantiate is selected based on the configuration class:

    • AlbertConfig configuration class: TFAlbertForSequenceClassification (ALBERT model)
    • BartConfig configuration class: TFBartForSequenceClassification (BART model)
    • BertConfig configuration class: TFBertForSequenceClassification (BERT model)
    • CTRLConfig configuration class: TFCTRLForSequenceClassification (CTRL model)
    • CamembertConfig configuration class: TFCamembertForSequenceClassification (CamemBERT model)
    • ConvBertConfig configuration class: TFConvBertForSequenceClassification (ConvBERT model)
    • DebertaConfig configuration class: TFDebertaForSequenceClassification (DeBERTa model)
    • DebertaV2Config configuration class: TFDebertaV2ForSequenceClassification (DeBERTa-v2 model)
    • DistilBertConfig configuration class: TFDistilBertForSequenceClassification (DistilBERT model)
    • ElectraConfig configuration class: TFElectraForSequenceClassification (ELECTRA model)
    • EsmConfig configuration class: TFEsmForSequenceClassification (ESM model)
    • FlaubertConfig configuration class: TFFlaubertForSequenceClassification (FlauBERT model)
    • FunnelConfig configuration class: TFFunnelForSequenceClassification (Funnel Transformer model)
    • GPT2Config configuration class: TFGPT2ForSequenceClassification (OpenAI GPT-2 model)
    • GPTJConfig configuration class: TFGPTJForSequenceClassification (GPT-J model)
    • LayoutLMConfig configuration class: TFLayoutLMForSequenceClassification (LayoutLM model)
    • LayoutLMv3Config configuration class: TFLayoutLMv3ForSequenceClassification (LayoutLMv3 model)
    • LongformerConfig configuration class: TFLongformerForSequenceClassification (Longformer model)
    • MPNetConfig configuration class: TFMPNetForSequenceClassification (MPNet model)
    • MobileBertConfig configuration class: TFMobileBertForSequenceClassification (MobileBERT model)
    • OpenAIGPTConfig configuration class: TFOpenAIGPTForSequenceClassification (OpenAI GPT model)
    • RemBertConfig configuration class: TFRemBertForSequenceClassification (RemBERT model)
    • RoFormerConfig configuration class: TFRoFormerForSequenceClassification (RoFormer model)
    • RobertaConfig configuration class: TFRobertaForSequenceClassification (RoBERTa model)
    • RobertaPreLayerNormConfig configuration class: TFRobertaPreLayerNormForSequenceClassification (RoBERTa-PreLayerNorm model)
    • TapasConfig configuration class: TFTapasForSequenceClassification (TAPAS model)
    • TransfoXLConfig configuration class: TFTransfoXLForSequenceClassification (Transformer-XL model)
    • XLMConfig configuration class: TFXLMForSequenceClassification (XLM model)
    • XLMRobertaConfig configuration class: TFXLMRobertaForSequenceClassification (XLM-RoBERTa model)
    • XLNetConfig configuration class: TFXLNetForSequenceClassification (XLNet model)
  • attn_implementation (str, optional) — The attention implementation to use in the model (if relevant). Can be any of "eager" (manual implementation of the attention), "sdpa" (using F.scaled_dot_product_attention), or "flash_attention_2" (using Dao-AILab/flash-attention). By default, if available, SDPA will be used for torch>=2.1.1. The default is otherwise the manual "eager" implementation.

Instantiates one of the model classes of the library (with a sequence classification head) from a configuration.

Note: Loading a model from its configuration file does not load the model weights. It only affects the model’s configuration. Use from_pretrained() to load the model weights.

Examples:

>>> from transformers import AutoConfig, TFAutoModelForSequenceClassification

>>> # Download configuration from huggingface.co and cache.
>>> config = AutoConfig.from_pretrained("google-bert/bert-base-cased")
>>> model = TFAutoModelForSequenceClassification.from_config(config)

from_pretrained

< >

( *model_args **kwargs )

Parameters

  • pretrained_model_name_or_path (str or os.PathLike) — Can be either:

    • A string, the model id of a pretrained model hosted inside a model repo on huggingface.co.
    • A path to a directory containing model weights saved using save_pretrained(), e.g., ./my_model_directory/.
    • A path or url to a PyTorch state_dict save file (e.g, ./pt_model/pytorch_model.bin). In this case, from_pt should be set to True and a configuration object should be provided as config argument. This loading path is slower than converting the PyTorch model in a TensorFlow model using the provided conversion scripts and loading the TensorFlow model afterwards.
  • model_args (additional positional arguments, optional) — Will be passed along to the underlying model __init__() method.
  • config (PretrainedConfig, optional) — Configuration for the model to use instead of an automatically loaded configuration. Configuration can be automatically loaded when:

    • The model is a model provided by the library (loaded with the model id string of a pretrained model).
    • The model was saved using save_pretrained() and is reloaded by supplying the save directory.
    • The model is loaded by supplying a local directory as pretrained_model_name_or_path and a configuration JSON file named config.json is found in the directory.
  • cache_dir (str or os.PathLike, optional) — Path to a directory in which a downloaded pretrained model configuration should be cached if the standard cache should not be used.
  • from_pt (bool, optional, defaults to False) — Load the model weights from a PyTorch checkpoint save file (see docstring of pretrained_model_name_or_path argument).
  • force_download (bool, optional, defaults to False) — Whether or not to force the (re-)download of the model weights and configuration files, overriding the cached versions if they exist.
  • resume_download (bool, optional, defaults to False) — Whether or not to delete incompletely received files. Will attempt to resume the download if such a file exists.
  • proxies (Dict[str, str], optional) — A dictionary of proxy servers to use by protocol or endpoint, e.g., {'http': 'foo.bar:3128', 'http://hostname': 'foo.bar:4012'}. The proxies are used on each request.
  • output_loading_info(bool, optional, defaults to False) — Whether ot not to also return a dictionary containing missing keys, unexpected keys and error messages.
  • local_files_only(bool, optional, defaults to False) — Whether or not to only look at local files (e.g., not try downloading the model).
  • revision (str, optional, defaults to "main") — The specific model version to use. It can be a branch name, a tag name, or a commit id, since we use a git-based system for storing models and other artifacts on huggingface.co, so revision can be any identifier allowed by git.
  • trust_remote_code (bool, optional, defaults to False) — Whether or not to allow for custom models defined on the Hub in their own modeling files. This option should only be set to True for repositories you trust and in which you have read the code, as it will execute code present on the Hub on your local machine.
  • code_revision (str, optional, defaults to "main") — The specific revision to use for the code on the Hub, if the code leaves in a different repository than the rest of the model. It can be a branch name, a tag name, or a commit id, since we use a git-based system for storing models and other artifacts on huggingface.co, so revision can be any identifier allowed by git.
  • kwargs (additional keyword arguments, optional) — Can be used to update the configuration object (after it being loaded) and initiate the model (e.g., output_attentions=True). Behaves differently depending on whether a config is provided or automatically loaded:

    • If a configuration is provided with config, **kwargs will be directly passed to the underlying model’s __init__ method (we assume all relevant updates to the configuration have already been done)
    • If a configuration is not provided, kwargs will be first passed to the configuration class initialization function (from_pretrained()). Each key of kwargs that corresponds to a configuration attribute will be used to override said attribute with the supplied kwargs value. Remaining keys that do not correspond to any configuration attribute will be passed to the underlying model’s __init__ function.

Instantiate one of the model classes of the library (with a sequence classification head) from a pretrained model.

The model class to instantiate is selected based on the model_type property of the config object (either passed as an argument or loaded from pretrained_model_name_or_path if possible), or when it’s missing, by falling back to using pattern matching on pretrained_model_name_or_path:

  • albert — TFAlbertForSequenceClassification (ALBERT model)
  • bart — TFBartForSequenceClassification (BART model)
  • bert — TFBertForSequenceClassification (BERT model)
  • camembert — TFCamembertForSequenceClassification (CamemBERT model)
  • convbert — TFConvBertForSequenceClassification (ConvBERT model)
  • ctrl — TFCTRLForSequenceClassification (CTRL model)
  • deberta — TFDebertaForSequenceClassification (DeBERTa model)
  • deberta-v2 — TFDebertaV2ForSequenceClassification (DeBERTa-v2 model)
  • distilbert — TFDistilBertForSequenceClassification (DistilBERT model)
  • electra — TFElectraForSequenceClassification (ELECTRA model)
  • esm — TFEsmForSequenceClassification (ESM model)
  • flaubert — TFFlaubertForSequenceClassification (FlauBERT model)
  • funnel — TFFunnelForSequenceClassification (Funnel Transformer model)
  • gpt-sw3 — TFGPT2ForSequenceClassification (GPT-Sw3 model)
  • gpt2 — TFGPT2ForSequenceClassification (OpenAI GPT-2 model)
  • gptj — TFGPTJForSequenceClassification (GPT-J model)
  • layoutlm — TFLayoutLMForSequenceClassification (LayoutLM model)
  • layoutlmv3 — TFLayoutLMv3ForSequenceClassification (LayoutLMv3 model)
  • longformer — TFLongformerForSequenceClassification (Longformer model)
  • mobilebert — TFMobileBertForSequenceClassification (MobileBERT model)
  • mpnet — TFMPNetForSequenceClassification (MPNet model)
  • openai-gpt — TFOpenAIGPTForSequenceClassification (OpenAI GPT model)
  • rembert — TFRemBertForSequenceClassification (RemBERT model)
  • roberta — TFRobertaForSequenceClassification (RoBERTa model)
  • roberta-prelayernorm — TFRobertaPreLayerNormForSequenceClassification (RoBERTa-PreLayerNorm model)
  • roformer — TFRoFormerForSequenceClassification (RoFormer model)
  • tapas — TFTapasForSequenceClassification (TAPAS model)
  • transfo-xl — TFTransfoXLForSequenceClassification (Transformer-XL model)
  • xlm — TFXLMForSequenceClassification (XLM model)
  • xlm-roberta — TFXLMRobertaForSequenceClassification (XLM-RoBERTa model)
  • xlnet — TFXLNetForSequenceClassification (XLNet model)

Examples:

>>> from transformers import AutoConfig, TFAutoModelForSequenceClassification

>>> # Download model and configuration from huggingface.co and cache.
>>> model = TFAutoModelForSequenceClassification.from_pretrained("google-bert/bert-base-cased")

>>> # Update configuration during loading
>>> model = TFAutoModelForSequenceClassification.from_pretrained("google-bert/bert-base-cased", output_attentions=True)
>>> model.config.output_attentions
True

>>> # Loading from a PyTorch checkpoint file instead of a TensorFlow model (slower)
>>> config = AutoConfig.from_pretrained("./pt_model/bert_pt_model_config.json")
>>> model = TFAutoModelForSequenceClassification.from_pretrained(
...     "./pt_model/bert_pytorch_model.bin", from_pt=True, config=config
... )

FlaxAutoModelForSequenceClassification

class transformers.FlaxAutoModelForSequenceClassification

< >

( *args **kwargs )

This is a generic model class that will be instantiated as one of the model classes of the library (with a sequence classification head) when created with the from_pretrained() class method or the from_config() class method.

This class cannot be instantiated directly using __init__() (throws an error).

from_config

< >

( **kwargs )

Parameters

  • config (PretrainedConfig) — The model class to instantiate is selected based on the configuration class:

    • AlbertConfig configuration class: FlaxAlbertForSequenceClassification (ALBERT model)
    • BartConfig configuration class: FlaxBartForSequenceClassification (BART model)
    • BertConfig configuration class: FlaxBertForSequenceClassification (BERT model)
    • BigBirdConfig configuration class: FlaxBigBirdForSequenceClassification (BigBird model)
    • DistilBertConfig configuration class: FlaxDistilBertForSequenceClassification (DistilBERT model)
    • ElectraConfig configuration class: FlaxElectraForSequenceClassification (ELECTRA model)
    • MBartConfig configuration class: FlaxMBartForSequenceClassification (mBART model)
    • RoFormerConfig configuration class: FlaxRoFormerForSequenceClassification (RoFormer model)
    • RobertaConfig configuration class: FlaxRobertaForSequenceClassification (RoBERTa model)
    • RobertaPreLayerNormConfig configuration class: FlaxRobertaPreLayerNormForSequenceClassification (RoBERTa-PreLayerNorm model)
    • XLMRobertaConfig configuration class: FlaxXLMRobertaForSequenceClassification (XLM-RoBERTa model)
  • attn_implementation (str, optional) — The attention implementation to use in the model (if relevant). Can be any of "eager" (manual implementation of the attention), "sdpa" (using F.scaled_dot_product_attention), or "flash_attention_2" (using Dao-AILab/flash-attention). By default, if available, SDPA will be used for torch>=2.1.1. The default is otherwise the manual "eager" implementation.

Instantiates one of the model classes of the library (with a sequence classification head) from a configuration.

Note: Loading a model from its configuration file does not load the model weights. It only affects the model’s configuration. Use from_pretrained() to load the model weights.

Examples:

>>> from transformers import AutoConfig, FlaxAutoModelForSequenceClassification

>>> # Download configuration from huggingface.co and cache.
>>> config = AutoConfig.from_pretrained("google-bert/bert-base-cased")
>>> model = FlaxAutoModelForSequenceClassification.from_config(config)

from_pretrained

< >

( *model_args **kwargs )

Parameters

  • pretrained_model_name_or_path (str or os.PathLike) — Can be either:

    • A string, the model id of a pretrained model hosted inside a model repo on huggingface.co.
    • A path to a directory containing model weights saved using save_pretrained(), e.g., ./my_model_directory/.
    • A path or url to a PyTorch state_dict save file (e.g, ./pt_model/pytorch_model.bin). In this case, from_pt should be set to True and a configuration object should be provided as config argument. This loading path is slower than converting the PyTorch model in a TensorFlow model using the provided conversion scripts and loading the TensorFlow model afterwards.
  • model_args (additional positional arguments, optional) — Will be passed along to the underlying model __init__() method.
  • config (PretrainedConfig, optional) — Configuration for the model to use instead of an automatically loaded configuration. Configuration can be automatically loaded when:

    • The model is a model provided by the library (loaded with the model id string of a pretrained model).
    • The model was saved using save_pretrained() and is reloaded by supplying the save directory.
    • The model is loaded by supplying a local directory as pretrained_model_name_or_path and a configuration JSON file named config.json is found in the directory.
  • cache_dir (str or os.PathLike, optional) — Path to a directory in which a downloaded pretrained model configuration should be cached if the standard cache should not be used.
  • from_pt (bool, optional, defaults to False) — Load the model weights from a PyTorch checkpoint save file (see docstring of pretrained_model_name_or_path argument).
  • force_download (bool, optional, defaults to False) — Whether or not to force the (re-)download of the model weights and configuration files, overriding the cached versions if they exist.
  • resume_download (bool, optional, defaults to False) — Whether or not to delete incompletely received files. Will attempt to resume the download if such a file exists.
  • proxies (Dict[str, str], optional) — A dictionary of proxy servers to use by protocol or endpoint, e.g., {'http': 'foo.bar:3128', 'http://hostname': 'foo.bar:4012'}. The proxies are used on each request.
  • output_loading_info(bool, optional, defaults to False) — Whether ot not to also return a dictionary containing missing keys, unexpected keys and error messages.
  • local_files_only(bool, optional, defaults to False) — Whether or not to only look at local files (e.g., not try downloading the model).
  • revision (str, optional, defaults to "main") — The specific model version to use. It can be a branch name, a tag name, or a commit id, since we use a git-based system for storing models and other artifacts on huggingface.co, so revision can be any identifier allowed by git.
  • trust_remote_code (bool, optional, defaults to False) — Whether or not to allow for custom models defined on the Hub in their own modeling files. This option should only be set to True for repositories you trust and in which you have read the code, as it will execute code present on the Hub on your local machine.
  • code_revision (str, optional, defaults to "main") — The specific revision to use for the code on the Hub, if the code leaves in a different repository than the rest of the model. It can be a branch name, a tag name, or a commit id, since we use a git-based system for storing models and other artifacts on huggingface.co, so revision can be any identifier allowed by git.
  • kwargs (additional keyword arguments, optional) — Can be used to update the configuration object (after it being loaded) and initiate the model (e.g., output_attentions=True). Behaves differently depending on whether a config is provided or automatically loaded:

    • If a configuration is provided with config, **kwargs will be directly passed to the underlying model’s __init__ method (we assume all relevant updates to the configuration have already been done)
    • If a configuration is not provided, kwargs will be first passed to the configuration class initialization function (from_pretrained()). Each key of kwargs that corresponds to a configuration attribute will be used to override said attribute with the supplied kwargs value. Remaining keys that do not correspond to any configuration attribute will be passed to the underlying model’s __init__ function.

Instantiate one of the model classes of the library (with a sequence classification head) from a pretrained model.

The model class to instantiate is selected based on the model_type property of the config object (either passed as an argument or loaded from pretrained_model_name_or_path if possible), or when it’s missing, by falling back to using pattern matching on pretrained_model_name_or_path:

  • albert — FlaxAlbertForSequenceClassification (ALBERT model)
  • bart — FlaxBartForSequenceClassification (BART model)
  • bert — FlaxBertForSequenceClassification (BERT model)
  • big_bird — FlaxBigBirdForSequenceClassification (BigBird model)
  • distilbert — FlaxDistilBertForSequenceClassification (DistilBERT model)
  • electra — FlaxElectraForSequenceClassification (ELECTRA model)
  • mbart — FlaxMBartForSequenceClassification (mBART model)
  • roberta — FlaxRobertaForSequenceClassification (RoBERTa model)
  • roberta-prelayernorm — FlaxRobertaPreLayerNormForSequenceClassification (RoBERTa-PreLayerNorm model)
  • roformer — FlaxRoFormerForSequenceClassification (RoFormer model)
  • xlm-roberta — FlaxXLMRobertaForSequenceClassification (XLM-RoBERTa model)

Examples:

>>> from transformers import AutoConfig, FlaxAutoModelForSequenceClassification

>>> # Download model and configuration from huggingface.co and cache.
>>> model = FlaxAutoModelForSequenceClassification.from_pretrained("google-bert/bert-base-cased")

>>> # Update configuration during loading
>>> model = FlaxAutoModelForSequenceClassification.from_pretrained("google-bert/bert-base-cased", output_attentions=True)
>>> model.config.output_attentions
True

>>> # Loading from a PyTorch checkpoint file instead of a TensorFlow model (slower)
>>> config = AutoConfig.from_pretrained("./pt_model/bert_pt_model_config.json")
>>> model = FlaxAutoModelForSequenceClassification.from_pretrained(
...     "./pt_model/bert_pytorch_model.bin", from_pt=True, config=config
... )

AutoModelForMultipleChoice

class transformers.AutoModelForMultipleChoice

< >

( *args **kwargs )

This is a generic model class that will be instantiated as one of the model classes of the library (with a multiple choice head) when created with the from_pretrained() class method or the from_config() class method.

This class cannot be instantiated directly using __init__() (throws an error).

from_config

< >

( **kwargs )

Parameters

  • config (PretrainedConfig) — The model class to instantiate is selected based on the configuration class:

    • AlbertConfig configuration class: AlbertForMultipleChoice (ALBERT model)
    • BertConfig configuration class: BertForMultipleChoice (BERT model)
    • BigBirdConfig configuration class: BigBirdForMultipleChoice (BigBird model)
    • CamembertConfig configuration class: CamembertForMultipleChoice (CamemBERT model)
    • CanineConfig configuration class: CanineForMultipleChoice (CANINE model)
    • ConvBertConfig configuration class: ConvBertForMultipleChoice (ConvBERT model)
    • Data2VecTextConfig configuration class: Data2VecTextForMultipleChoice (Data2VecText model)
    • DebertaV2Config configuration class: DebertaV2ForMultipleChoice (DeBERTa-v2 model)
    • DistilBertConfig configuration class: DistilBertForMultipleChoice (DistilBERT model)
    • ElectraConfig configuration class: ElectraForMultipleChoice (ELECTRA model)
    • ErnieConfig configuration class: ErnieForMultipleChoice (ERNIE model)
    • ErnieMConfig configuration class: ErnieMForMultipleChoice (ErnieM model)
    • FNetConfig configuration class: FNetForMultipleChoice (FNet model)
    • FlaubertConfig configuration class: FlaubertForMultipleChoice (FlauBERT model)
    • FunnelConfig configuration class: FunnelForMultipleChoice (Funnel Transformer model)
    • IBertConfig configuration class: IBertForMultipleChoice (I-BERT model)
    • LongformerConfig configuration class: LongformerForMultipleChoice (Longformer model)
    • LukeConfig configuration class: LukeForMultipleChoice (LUKE model)
    • MPNetConfig configuration class: MPNetForMultipleChoice (MPNet model)
    • MegaConfig configuration class: MegaForMultipleChoice (MEGA model)
    • MegatronBertConfig configuration class: MegatronBertForMultipleChoice (Megatron-BERT model)
    • MobileBertConfig configuration class: MobileBertForMultipleChoice (MobileBERT model)
    • MraConfig configuration class: MraForMultipleChoice (MRA model)
    • NezhaConfig configuration class: NezhaForMultipleChoice (Nezha model)
    • NystromformerConfig configuration class: NystromformerForMultipleChoice (Nyströmformer model)
    • QDQBertConfig configuration class: QDQBertForMultipleChoice (QDQBert model)
    • RemBertConfig configuration class: RemBertForMultipleChoice (RemBERT model)
    • RoCBertConfig configuration class: RoCBertForMultipleChoice (RoCBert model)
    • RoFormerConfig configuration class: RoFormerForMultipleChoice (RoFormer model)
    • RobertaConfig configuration class: RobertaForMultipleChoice (RoBERTa model)
    • RobertaPreLayerNormConfig configuration class: RobertaPreLayerNormForMultipleChoice (RoBERTa-PreLayerNorm model)
    • SqueezeBertConfig configuration class: SqueezeBertForMultipleChoice (SqueezeBERT model)
    • XLMConfig configuration class: XLMForMultipleChoice (XLM model)
    • XLMRobertaConfig configuration class: XLMRobertaForMultipleChoice (XLM-RoBERTa model)
    • XLMRobertaXLConfig configuration class: XLMRobertaXLForMultipleChoice (XLM-RoBERTa-XL model)
    • XLNetConfig configuration class: XLNetForMultipleChoice (XLNet model)
    • XmodConfig configuration class: XmodForMultipleChoice (X-MOD model)
    • YosoConfig configuration class: YosoForMultipleChoice (YOSO model)
  • attn_implementation (str, optional) — The attention implementation to use in the model (if relevant). Can be any of "eager" (manual implementation of the attention), "sdpa" (using F.scaled_dot_product_attention), or "flash_attention_2" (using Dao-AILab/flash-attention). By default, if available, SDPA will be used for torch>=2.1.1. The default is otherwise the manual "eager" implementation.

Instantiates one of the model classes of the library (with a multiple choice head) from a configuration.

Note: Loading a model from its configuration file does not load the model weights. It only affects the model’s configuration. Use from_pretrained() to load the model weights.

Examples:

>>> from transformers import AutoConfig, AutoModelForMultipleChoice

>>> # Download configuration from huggingface.co and cache.
>>> config = AutoConfig.from_pretrained("google-bert/bert-base-cased")
>>> model = AutoModelForMultipleChoice.from_config(config)

from_pretrained

< >

( *model_args **kwargs )

Parameters

  • pretrained_model_name_or_path (str or os.PathLike) — Can be either:

    • A string, the model id of a pretrained model hosted inside a model repo on huggingface.co.
    • A path to a directory containing model weights saved using save_pretrained(), e.g., ./my_model_directory/.
    • A path or url to a tensorflow index checkpoint file (e.g, ./tf_model/model.ckpt.index). In this case, from_tf should be set to True and a configuration object should be provided as config argument. This loading path is slower than converting the TensorFlow checkpoint in a PyTorch model using the provided conversion scripts and loading the PyTorch model afterwards.
  • model_args (additional positional arguments, optional) — Will be passed along to the underlying model __init__() method.
  • config (PretrainedConfig, optional) — Configuration for the model to use instead of an automatically loaded configuration. Configuration can be automatically loaded when:

    • The model is a model provided by the library (loaded with the model id string of a pretrained model).
    • The model was saved using save_pretrained() and is reloaded by supplying the save directory.
    • The model is loaded by supplying a local directory as pretrained_model_name_or_path and a configuration JSON file named config.json is found in the directory.
  • state_dict (Dict[str, torch.Tensor], optional) — A state dictionary to use instead of a state dictionary loaded from saved weights file.

    This option can be used if you want to create a model from a pretrained configuration but load your own weights. In this case though, you should check if using save_pretrained() and from_pretrained() is not a simpler option.

  • cache_dir (str or os.PathLike, optional) — Path to a directory in which a downloaded pretrained model configuration should be cached if the standard cache should not be used.
  • from_tf (bool, optional, defaults to False) — Load the model weights from a TensorFlow checkpoint save file (see docstring of pretrained_model_name_or_path argument).
  • force_download (bool, optional, defaults to False) — Whether or not to force the (re-)download of the model weights and configuration files, overriding the cached versions if they exist.
  • resume_download (bool, optional, defaults to False) — Whether or not to delete incompletely received files. Will attempt to resume the download if such a file exists.
  • proxies (Dict[str, str], optional) — A dictionary of proxy servers to use by protocol or endpoint, e.g., {'http': 'foo.bar:3128', 'http://hostname': 'foo.bar:4012'}. The proxies are used on each request.
  • output_loading_info(bool, optional, defaults to False) — Whether ot not to also return a dictionary containing missing keys, unexpected keys and error messages.
  • local_files_only(bool, optional, defaults to False) — Whether or not to only look at local files (e.g., not try downloading the model).
  • revision (str, optional, defaults to "main") — The specific model version to use. It can be a branch name, a tag name, or a commit id, since we use a git-based system for storing models and other artifacts on huggingface.co, so revision can be any identifier allowed by git.
  • trust_remote_code (bool, optional, defaults to False) — Whether or not to allow for custom models defined on the Hub in their own modeling files. This option should only be set to True for repositories you trust and in which you have read the code, as it will execute code present on the Hub on your local machine.
  • code_revision (str, optional, defaults to "main") — The specific revision to use for the code on the Hub, if the code leaves in a different repository than the rest of the model. It can be a branch name, a tag name, or a commit id, since we use a git-based system for storing models and other artifacts on huggingface.co, so revision can be any identifier allowed by git.
  • kwargs (additional keyword arguments, optional) — Can be used to update the configuration object (after it being loaded) and initiate the model (e.g., output_attentions=True). Behaves differently depending on whether a config is provided or automatically loaded:

    • If a configuration is provided with config, **kwargs will be directly passed to the underlying model’s __init__ method (we assume all relevant updates to the configuration have already been done)
    • If a configuration is not provided, kwargs will be first passed to the configuration class initialization function (from_pretrained()). Each key of kwargs that corresponds to a configuration attribute will be used to override said attribute with the supplied kwargs value. Remaining keys that do not correspond to any configuration attribute will be passed to the underlying model’s __init__ function.

Instantiate one of the model classes of the library (with a multiple choice head) from a pretrained model.

The model class to instantiate is selected based on the model_type property of the config object (either passed as an argument or loaded from pretrained_model_name_or_path if possible), or when it’s missing, by falling back to using pattern matching on pretrained_model_name_or_path:

  • albert — AlbertForMultipleChoice (ALBERT model)
  • bert — BertForMultipleChoice (BERT model)
  • big_bird — BigBirdForMultipleChoice (BigBird model)
  • camembert — CamembertForMultipleChoice (CamemBERT model)
  • canine — CanineForMultipleChoice (CANINE model)
  • convbert — ConvBertForMultipleChoice (ConvBERT model)
  • data2vec-text — Data2VecTextForMultipleChoice (Data2VecText model)
  • deberta-v2 — DebertaV2ForMultipleChoice (DeBERTa-v2 model)
  • distilbert — DistilBertForMultipleChoice (DistilBERT model)
  • electra — ElectraForMultipleChoice (ELECTRA model)
  • ernie — ErnieForMultipleChoice (ERNIE model)
  • ernie_m — ErnieMForMultipleChoice (ErnieM model)
  • flaubert — FlaubertForMultipleChoice (FlauBERT model)
  • fnet — FNetForMultipleChoice (FNet model)
  • funnel — FunnelForMultipleChoice (Funnel Transformer model)
  • ibert — IBertForMultipleChoice (I-BERT model)
  • longformer — LongformerForMultipleChoice (Longformer model)
  • luke — LukeForMultipleChoice (LUKE model)
  • mega — MegaForMultipleChoice (MEGA model)
  • megatron-bert — MegatronBertForMultipleChoice (Megatron-BERT model)
  • mobilebert — MobileBertForMultipleChoice (MobileBERT model)
  • mpnet — MPNetForMultipleChoice (MPNet model)
  • mra — MraForMultipleChoice (MRA model)
  • nezha — NezhaForMultipleChoice (Nezha model)
  • nystromformer — NystromformerForMultipleChoice (Nyströmformer model)
  • qdqbert — QDQBertForMultipleChoice (QDQBert model)
  • rembert — RemBertForMultipleChoice (RemBERT model)
  • roberta — RobertaForMultipleChoice (RoBERTa model)
  • roberta-prelayernorm — RobertaPreLayerNormForMultipleChoice (RoBERTa-PreLayerNorm model)
  • roc_bert — RoCBertForMultipleChoice (RoCBert model)
  • roformer — RoFormerForMultipleChoice (RoFormer model)
  • squeezebert — SqueezeBertForMultipleChoice (SqueezeBERT model)
  • xlm — XLMForMultipleChoice (XLM model)
  • xlm-roberta — XLMRobertaForMultipleChoice (XLM-RoBERTa model)
  • xlm-roberta-xl — XLMRobertaXLForMultipleChoice (XLM-RoBERTa-XL model)
  • xlnet — XLNetForMultipleChoice (XLNet model)
  • xmod — XmodForMultipleChoice (X-MOD model)
  • yoso — YosoForMultipleChoice (YOSO model)

The model is set in evaluation mode by default using model.eval() (so for instance, dropout modules are deactivated). To train the model, you should first set it back in training mode with model.train()

Examples:

>>> from transformers import AutoConfig, AutoModelForMultipleChoice

>>> # Download model and configuration from huggingface.co and cache.
>>> model = AutoModelForMultipleChoice.from_pretrained("google-bert/bert-base-cased")

>>> # Update configuration during loading
>>> model = AutoModelForMultipleChoice.from_pretrained("google-bert/bert-base-cased", output_attentions=True)
>>> model.config.output_attentions
True

>>> # Loading from a TF checkpoint file instead of a PyTorch model (slower)
>>> config = AutoConfig.from_pretrained("./tf_model/bert_tf_model_config.json")
>>> model = AutoModelForMultipleChoice.from_pretrained(
...     "./tf_model/bert_tf_checkpoint.ckpt.index", from_tf=True, config=config
... )

TFAutoModelForMultipleChoice

class transformers.TFAutoModelForMultipleChoice

< >

( *args **kwargs )

This is a generic model class that will be instantiated as one of the model classes of the library (with a multiple choice head) when created with the from_pretrained() class method or the from_config() class method.

This class cannot be instantiated directly using __init__() (throws an error).

from_config

< >

( **kwargs )

Parameters

  • config (PretrainedConfig) — The model class to instantiate is selected based on the configuration class:

    • AlbertConfig configuration class: TFAlbertForMultipleChoice (ALBERT model)
    • BertConfig configuration class: TFBertForMultipleChoice (BERT model)
    • CamembertConfig configuration class: TFCamembertForMultipleChoice (CamemBERT model)
    • ConvBertConfig configuration class: TFConvBertForMultipleChoice (ConvBERT model)
    • DebertaV2Config configuration class: TFDebertaV2ForMultipleChoice (DeBERTa-v2 model)
    • DistilBertConfig configuration class: TFDistilBertForMultipleChoice (DistilBERT model)
    • ElectraConfig configuration class: TFElectraForMultipleChoice (ELECTRA model)
    • FlaubertConfig configuration class: TFFlaubertForMultipleChoice (FlauBERT model)
    • FunnelConfig configuration class: TFFunnelForMultipleChoice (Funnel Transformer model)
    • LongformerConfig configuration class: TFLongformerForMultipleChoice (Longformer model)
    • MPNetConfig configuration class: TFMPNetForMultipleChoice (MPNet model)
    • MobileBertConfig configuration class: TFMobileBertForMultipleChoice (MobileBERT model)
    • RemBertConfig configuration class: TFRemBertForMultipleChoice (RemBERT model)
    • RoFormerConfig configuration class: TFRoFormerForMultipleChoice (RoFormer model)
    • RobertaConfig configuration class: TFRobertaForMultipleChoice (RoBERTa model)
    • RobertaPreLayerNormConfig configuration class: TFRobertaPreLayerNormForMultipleChoice (RoBERTa-PreLayerNorm model)
    • XLMConfig configuration class: TFXLMForMultipleChoice (XLM model)
    • XLMRobertaConfig configuration class: TFXLMRobertaForMultipleChoice (XLM-RoBERTa model)
    • XLNetConfig configuration class: TFXLNetForMultipleChoice (XLNet model)
  • attn_implementation (str, optional) — The attention implementation to use in the model (if relevant). Can be any of "eager" (manual implementation of the attention), "sdpa" (using F.scaled_dot_product_attention), or "flash_attention_2" (using Dao-AILab/flash-attention). By default, if available, SDPA will be used for torch>=2.1.1. The default is otherwise the manual "eager" implementation.

Instantiates one of the model classes of the library (with a multiple choice head) from a configuration.

Note: Loading a model from its configuration file does not load the model weights. It only affects the model’s configuration. Use from_pretrained() to load the model weights.

Examples:

>>> from transformers import AutoConfig, TFAutoModelForMultipleChoice

>>> # Download configuration from huggingface.co and cache.
>>> config = AutoConfig.from_pretrained("google-bert/bert-base-cased")
>>> model = TFAutoModelForMultipleChoice.from_config(config)

from_pretrained

< >

( *model_args **kwargs )

Parameters

  • pretrained_model_name_or_path (str or os.PathLike) — Can be either:

    • A string, the model id of a pretrained model hosted inside a model repo on huggingface.co.
    • A path to a directory containing model weights saved using save_pretrained(), e.g., ./my_model_directory/.
    • A path or url to a PyTorch state_dict save file (e.g, ./pt_model/pytorch_model.bin). In this case, from_pt should be set to True and a configuration object should be provided as config argument. This loading path is slower than converting the PyTorch model in a TensorFlow model using the provided conversion scripts and loading the TensorFlow model afterwards.
  • model_args (additional positional arguments, optional) — Will be passed along to the underlying model __init__() method.
  • config (PretrainedConfig, optional) — Configuration for the model to use instead of an automatically loaded configuration. Configuration can be automatically loaded when:

    • The model is a model provided by the library (loaded with the model id string of a pretrained model).
    • The model was saved using save_pretrained() and is reloaded by supplying the save directory.
    • The model is loaded by supplying a local directory as pretrained_model_name_or_path and a configuration JSON file named config.json is found in the directory.
  • cache_dir (str or os.PathLike, optional) — Path to a directory in which a downloaded pretrained model configuration should be cached if the standard cache should not be used.
  • from_pt (bool, optional, defaults to False) — Load the model weights from a PyTorch checkpoint save file (see docstring of pretrained_model_name_or_path argument).
  • force_download (bool, optional, defaults to False) — Whether or not to force the (re-)download of the model weights and configuration files, overriding the cached versions if they exist.
  • resume_download (bool, optional, defaults to False) — Whether or not to delete incompletely received files. Will attempt to resume the download if such a file exists.
  • proxies (Dict[str, str], optional) — A dictionary of proxy servers to use by protocol or endpoint, e.g., {'http': 'foo.bar:3128', 'http://hostname': 'foo.bar:4012'}. The proxies are used on each request.
  • output_loading_info(bool, optional, defaults to False) — Whether ot not to also return a dictionary containing missing keys, unexpected keys and error messages.
  • local_files_only(bool, optional, defaults to False) — Whether or not to only look at local files (e.g., not try downloading the model).
  • revision (str, optional, defaults to "main") — The specific model version to use. It can be a branch name, a tag name, or a commit id, since we use a git-based system for storing models and other artifacts on huggingface.co, so revision can be any identifier allowed by git.
  • trust_remote_code (bool, optional, defaults to False) — Whether or not to allow for custom models defined on the Hub in their own modeling files. This option should only be set to True for repositories you trust and in which you have read the code, as it will execute code present on the Hub on your local machine.
  • code_revision (str, optional, defaults to "main") — The specific revision to use for the code on the Hub, if the code leaves in a different repository than the rest of the model. It can be a branch name, a tag name, or a commit id, since we use a git-based system for storing models and other artifacts on huggingface.co, so revision can be any identifier allowed by git.
  • kwargs (additional keyword arguments, optional) — Can be used to update the configuration object (after it being loaded) and initiate the model (e.g., output_attentions=True). Behaves differently depending on whether a config is provided or automatically loaded:

    • If a configuration is provided with config, **kwargs will be directly passed to the underlying model’s __init__ method (we assume all relevant updates to the configuration have already been done)
    • If a configuration is not provided, kwargs will be first passed to the configuration class initialization function (from_pretrained()). Each key of kwargs that corresponds to a configuration attribute will be used to override said attribute with the supplied kwargs value. Remaining keys that do not correspond to any configuration attribute will be passed to the underlying model’s __init__ function.

Instantiate one of the model classes of the library (with a multiple choice head) from a pretrained model.

The model class to instantiate is selected based on the model_type property of the config object (either passed as an argument or loaded from pretrained_model_name_or_path if possible), or when it’s missing, by falling back to using pattern matching on pretrained_model_name_or_path:

  • albert — TFAlbertForMultipleChoice (ALBERT model)
  • bert — TFBertForMultipleChoice (BERT model)
  • camembert — TFCamembertForMultipleChoice (CamemBERT model)
  • convbert — TFConvBertForMultipleChoice (ConvBERT model)
  • deberta-v2 — TFDebertaV2ForMultipleChoice (DeBERTa-v2 model)
  • distilbert — TFDistilBertForMultipleChoice (DistilBERT model)
  • electra — TFElectraForMultipleChoice (ELECTRA model)
  • flaubert — TFFlaubertForMultipleChoice (FlauBERT model)
  • funnel — TFFunnelForMultipleChoice (Funnel Transformer model)
  • longformer — TFLongformerForMultipleChoice (Longformer model)
  • mobilebert — TFMobileBertForMultipleChoice (MobileBERT model)
  • mpnet — TFMPNetForMultipleChoice (MPNet model)
  • rembert — TFRemBertForMultipleChoice (RemBERT model)
  • roberta — TFRobertaForMultipleChoice (RoBERTa model)
  • roberta-prelayernorm — TFRobertaPreLayerNormForMultipleChoice (RoBERTa-PreLayerNorm model)
  • roformer — TFRoFormerForMultipleChoice (RoFormer model)
  • xlm — TFXLMForMultipleChoice (XLM model)
  • xlm-roberta — TFXLMRobertaForMultipleChoice (XLM-RoBERTa model)
  • xlnet — TFXLNetForMultipleChoice (XLNet model)

Examples:

>>> from transformers import AutoConfig, TFAutoModelForMultipleChoice

>>> # Download model and configuration from huggingface.co and cache.
>>> model = TFAutoModelForMultipleChoice.from_pretrained("google-bert/bert-base-cased")

>>> # Update configuration during loading
>>> model = TFAutoModelForMultipleChoice.from_pretrained("google-bert/bert-base-cased", output_attentions=True)
>>> model.config.output_attentions
True

>>> # Loading from a PyTorch checkpoint file instead of a TensorFlow model (slower)
>>> config = AutoConfig.from_pretrained("./pt_model/bert_pt_model_config.json")
>>> model = TFAutoModelForMultipleChoice.from_pretrained(
...     "./pt_model/bert_pytorch_model.bin", from_pt=True, config=config
... )

FlaxAutoModelForMultipleChoice

class transformers.FlaxAutoModelForMultipleChoice

< >

( *args **kwargs )

This is a generic model class that will be instantiated as one of the model classes of the library (with a multiple choice head) when created with the from_pretrained() class method or the from_config() class method.

This class cannot be instantiated directly using __init__() (throws an error).

from_config

< >

( **kwargs )

Parameters

  • config (PretrainedConfig) — The model class to instantiate is selected based on the configuration class:

    • AlbertConfig configuration class: FlaxAlbertForMultipleChoice (ALBERT model)
    • BertConfig configuration class: FlaxBertForMultipleChoice (BERT model)
    • BigBirdConfig configuration class: FlaxBigBirdForMultipleChoice (BigBird model)
    • DistilBertConfig configuration class: FlaxDistilBertForMultipleChoice (DistilBERT model)
    • ElectraConfig configuration class: FlaxElectraForMultipleChoice (ELECTRA model)
    • RoFormerConfig configuration class: FlaxRoFormerForMultipleChoice (RoFormer model)
    • RobertaConfig configuration class: FlaxRobertaForMultipleChoice (RoBERTa model)
    • RobertaPreLayerNormConfig configuration class: FlaxRobertaPreLayerNormForMultipleChoice (RoBERTa-PreLayerNorm model)
    • XLMRobertaConfig configuration class: FlaxXLMRobertaForMultipleChoice (XLM-RoBERTa model)
  • attn_implementation (str, optional) — The attention implementation to use in the model (if relevant). Can be any of "eager" (manual implementation of the attention), "sdpa" (using F.scaled_dot_product_attention), or "flash_attention_2" (using Dao-AILab/flash-attention). By default, if available, SDPA will be used for torch>=2.1.1. The default is otherwise the manual "eager" implementation.

Instantiates one of the model classes of the library (with a multiple choice head) from a configuration.

Note: Loading a model from its configuration file does not load the model weights. It only affects the model’s configuration. Use from_pretrained() to load the model weights.

Examples:

>>> from transformers import AutoConfig, FlaxAutoModelForMultipleChoice

>>> # Download configuration from huggingface.co and cache.
>>> config = AutoConfig.from_pretrained("google-bert/bert-base-cased")
>>> model = FlaxAutoModelForMultipleChoice.from_config(config)

from_pretrained

< >

( *model_args **kwargs )

Parameters

  • pretrained_model_name_or_path (str or os.PathLike) — Can be either:

    • A string, the model id of a pretrained model hosted inside a model repo on huggingface.co.
    • A path to a directory containing model weights saved using save_pretrained(), e.g., ./my_model_directory/.
    • A path or url to a PyTorch state_dict save file (e.g, ./pt_model/pytorch_model.bin). In this case, from_pt should be set to True and a configuration object should be provided as config argument. This loading path is slower than converting the PyTorch model in a TensorFlow model using the provided conversion scripts and loading the TensorFlow model afterwards.
  • model_args (additional positional arguments, optional) — Will be passed along to the underlying model __init__() method.
  • config (PretrainedConfig, optional) — Configuration for the model to use instead of an automatically loaded configuration. Configuration can be automatically loaded when:

    • The model is a model provided by the library (loaded with the model id string of a pretrained model).
    • The model was saved using save_pretrained() and is reloaded by supplying the save directory.
    • The model is loaded by supplying a local directory as pretrained_model_name_or_path and a configuration JSON file named config.json is found in the directory.
  • cache_dir (str or os.PathLike, optional) — Path to a directory in which a downloaded pretrained model configuration should be cached if the standard cache should not be used.
  • from_pt (bool, optional, defaults to False) — Load the model weights from a PyTorch checkpoint save file (see docstring of pretrained_model_name_or_path argument).
  • force_download (bool, optional, defaults to False) — Whether or not to force the (re-)download of the model weights and configuration files, overriding the cached versions if they exist.
  • resume_download (bool, optional, defaults to False) — Whether or not to delete incompletely received files. Will attempt to resume the download if such a file exists.
  • proxies (Dict[str, str], optional) — A dictionary of proxy servers to use by protocol or endpoint, e.g., {'http': 'foo.bar:3128', 'http://hostname': 'foo.bar:4012'}. The proxies are used on each request.
  • output_loading_info(bool, optional, defaults to False) — Whether ot not to also return a dictionary containing missing keys, unexpected keys and error messages.
  • local_files_only(bool, optional, defaults to False) — Whether or not to only look at local files (e.g., not try downloading the model).
  • revision (str, optional, defaults to "main") — The specific model version to use. It can be a branch name, a tag name, or a commit id, since we use a git-based system for storing models and other artifacts on huggingface.co, so revision can be any identifier allowed by git.
  • trust_remote_code (bool, optional, defaults to False) — Whether or not to allow for custom models defined on the Hub in their own modeling files. This option should only be set to True for repositories you trust and in which you have read the code, as it will execute code present on the Hub on your local machine.
  • code_revision (str, optional, defaults to "main") — The specific revision to use for the code on the Hub, if the code leaves in a different repository than the rest of the model. It can be a branch name, a tag name, or a commit id, since we use a git-based system for storing models and other artifacts on huggingface.co, so revision can be any identifier allowed by git.
  • kwargs (additional keyword arguments, optional) — Can be used to update the configuration object (after it being loaded) and initiate the model (e.g., output_attentions=True). Behaves differently depending on whether a config is provided or automatically loaded:

    • If a configuration is provided with config, **kwargs will be directly passed to the underlying model’s __init__ method (we assume all relevant updates to the configuration have already been done)
    • If a configuration is not provided, kwargs will be first passed to the configuration class initialization function (from_pretrained()). Each key of kwargs that corresponds to a configuration attribute will be used to override said attribute with the supplied kwargs value. Remaining keys that do not correspond to any configuration attribute will be passed to the underlying model’s __init__ function.

Instantiate one of the model classes of the library (with a multiple choice head) from a pretrained model.

The model class to instantiate is selected based on the model_type property of the config object (either passed as an argument or loaded from pretrained_model_name_or_path if possible), or when it’s missing, by falling back to using pattern matching on pretrained_model_name_or_path:

  • albert — FlaxAlbertForMultipleChoice (ALBERT model)
  • bert — FlaxBertForMultipleChoice (BERT model)
  • big_bird — FlaxBigBirdForMultipleChoice (BigBird model)
  • distilbert — FlaxDistilBertForMultipleChoice (DistilBERT model)
  • electra — FlaxElectraForMultipleChoice (ELECTRA model)
  • roberta — FlaxRobertaForMultipleChoice (RoBERTa model)
  • roberta-prelayernorm — FlaxRobertaPreLayerNormForMultipleChoice (RoBERTa-PreLayerNorm model)
  • roformer — FlaxRoFormerForMultipleChoice (RoFormer model)
  • xlm-roberta — FlaxXLMRobertaForMultipleChoice (XLM-RoBERTa model)

Examples:

>>> from transformers import AutoConfig, FlaxAutoModelForMultipleChoice

>>> # Download model and configuration from huggingface.co and cache.
>>> model = FlaxAutoModelForMultipleChoice.from_pretrained("google-bert/bert-base-cased")

>>> # Update configuration during loading
>>> model = FlaxAutoModelForMultipleChoice.from_pretrained("google-bert/bert-base-cased", output_attentions=True)
>>> model.config.output_attentions
True

>>> # Loading from a PyTorch checkpoint file instead of a TensorFlow model (slower)
>>> config = AutoConfig.from_pretrained("./pt_model/bert_pt_model_config.json")
>>> model = FlaxAutoModelForMultipleChoice.from_pretrained(
...     "./pt_model/bert_pytorch_model.bin", from_pt=True, config=config
... )

AutoModelForNextSentencePrediction

class transformers.AutoModelForNextSentencePrediction

< >

( *args **kwargs )

This is a generic model class that will be instantiated as one of the model classes of the library (with a next sentence prediction head) when created with the from_pretrained() class method or the from_config() class method.

This class cannot be instantiated directly using __init__() (throws an error).

from_config

< >

( **kwargs )

Parameters

  • config (PretrainedConfig) — The model class to instantiate is selected based on the configuration class:

    • BertConfig configuration class: BertForNextSentencePrediction (BERT model)
    • ErnieConfig configuration class: ErnieForNextSentencePrediction (ERNIE model)
    • FNetConfig configuration class: FNetForNextSentencePrediction (FNet model)
    • MegatronBertConfig configuration class: MegatronBertForNextSentencePrediction (Megatron-BERT model)
    • MobileBertConfig configuration class: MobileBertForNextSentencePrediction (MobileBERT model)
    • NezhaConfig configuration class: NezhaForNextSentencePrediction (Nezha model)
    • QDQBertConfig configuration class: QDQBertForNextSentencePrediction (QDQBert model)
  • attn_implementation (str, optional) — The attention implementation to use in the model (if relevant). Can be any of "eager" (manual implementation of the attention), "sdpa" (using F.scaled_dot_product_attention), or "flash_attention_2" (using Dao-AILab/flash-attention). By default, if available, SDPA will be used for torch>=2.1.1. The default is otherwise the manual "eager" implementation.

Instantiates one of the model classes of the library (with a next sentence prediction head) from a configuration.

Note: Loading a model from its configuration file does not load the model weights. It only affects the model’s configuration. Use from_pretrained() to load the model weights.

Examples:

>>> from transformers import AutoConfig, AutoModelForNextSentencePrediction

>>> # Download configuration from huggingface.co and cache.
>>> config = AutoConfig.from_pretrained("google-bert/bert-base-cased")
>>> model = AutoModelForNextSentencePrediction.from_config(config)

from_pretrained

< >

( *model_args **kwargs )

Parameters

  • pretrained_model_name_or_path (str or os.PathLike) — Can be either:

    • A string, the model id of a pretrained model hosted inside a model repo on huggingface.co.
    • A path to a directory containing model weights saved using save_pretrained(), e.g., ./my_model_directory/.
    • A path or url to a tensorflow index checkpoint file (e.g, ./tf_model/model.ckpt.index). In this case, from_tf should be set to True and a configuration object should be provided as config argument. This loading path is slower than converting the TensorFlow checkpoint in a PyTorch model using the provided conversion scripts and loading the PyTorch model afterwards.
  • model_args (additional positional arguments, optional) — Will be passed along to the underlying model __init__() method.
  • config (PretrainedConfig, optional) — Configuration for the model to use instead of an automatically loaded configuration. Configuration can be automatically loaded when:

    • The model is a model provided by the library (loaded with the model id string of a pretrained model).
    • The model was saved using save_pretrained() and is reloaded by supplying the save directory.
    • The model is loaded by supplying a local directory as pretrained_model_name_or_path and a configuration JSON file named config.json is found in the directory.
  • state_dict (Dict[str, torch.Tensor], optional) — A state dictionary to use instead of a state dictionary loaded from saved weights file.

    This option can be used if you want to create a model from a pretrained configuration but load your own weights. In this case though, you should check if using save_pretrained() and from_pretrained() is not a simpler option.

  • cache_dir (str or os.PathLike, optional) — Path to a directory in which a downloaded pretrained model configuration should be cached if the standard cache should not be used.
  • from_tf (bool, optional, defaults to False) — Load the model weights from a TensorFlow checkpoint save file (see docstring of pretrained_model_name_or_path argument).
  • force_download (bool, optional, defaults to False) — Whether or not to force the (re-)download of the model weights and configuration files, overriding the cached versions if they exist.
  • resume_download (bool, optional, defaults to False) — Whether or not to delete incompletely received files. Will attempt to resume the download if such a file exists.
  • proxies (Dict[str, str], optional) — A dictionary of proxy servers to use by protocol or endpoint, e.g., {'http': 'foo.bar:3128', 'http://hostname': 'foo.bar:4012'}. The proxies are used on each request.
  • output_loading_info(bool, optional, defaults to False) — Whether ot not to also return a dictionary containing missing keys, unexpected keys and error messages.
  • local_files_only(bool, optional, defaults to False) — Whether or not to only look at local files (e.g., not try downloading the model).
  • revision (str, optional, defaults to "main") — The specific model version to use. It can be a branch name, a tag name, or a commit id, since we use a git-based system for storing models and other artifacts on huggingface.co, so revision can be any identifier allowed by git.
  • trust_remote_code (bool, optional, defaults to False) — Whether or not to allow for custom models defined on the Hub in their own modeling files. This option should only be set to True for repositories you trust and in which you have read the code, as it will execute code present on the Hub on your local machine.
  • code_revision (str, optional, defaults to "main") — The specific revision to use for the code on the Hub, if the code leaves in a different repository than the rest of the model. It can be a branch name, a tag name, or a commit id, since we use a git-based system for storing models and other artifacts on huggingface.co, so revision can be any identifier allowed by git.
  • kwargs (additional keyword arguments, optional) — Can be used to update the configuration object (after it being loaded) and initiate the model (e.g., output_attentions=True). Behaves differently depending on whether a config is provided or automatically loaded:

    • If a configuration is provided with config, **kwargs will be directly passed to the underlying model’s __init__ method (we assume all relevant updates to the configuration have already been done)
    • If a configuration is not provided, kwargs will be first passed to the configuration class initialization function (from_pretrained()). Each key of kwargs that corresponds to a configuration attribute will be used to override said attribute with the supplied kwargs value. Remaining keys that do not correspond to any configuration attribute will be passed to the underlying model’s __init__ function.

Instantiate one of the model classes of the library (with a next sentence prediction head) from a pretrained model.

The model class to instantiate is selected based on the model_type property of the config object (either passed as an argument or loaded from pretrained_model_name_or_path if possible), or when it’s missing, by falling back to using pattern matching on pretrained_model_name_or_path:

  • bert — BertForNextSentencePrediction (BERT model)
  • ernie — ErnieForNextSentencePrediction (ERNIE model)
  • fnet — FNetForNextSentencePrediction (FNet model)
  • megatron-bert — MegatronBertForNextSentencePrediction (Megatron-BERT model)
  • mobilebert — MobileBertForNextSentencePrediction (MobileBERT model)
  • nezha — NezhaForNextSentencePrediction (Nezha model)
  • qdqbert — QDQBertForNextSentencePrediction (QDQBert model)

The model is set in evaluation mode by default using model.eval() (so for instance, dropout modules are deactivated). To train the model, you should first set it back in training mode with model.train()

Examples:

>>> from transformers import AutoConfig, AutoModelForNextSentencePrediction

>>> # Download model and configuration from huggingface.co and cache.
>>> model = AutoModelForNextSentencePrediction.from_pretrained("google-bert/bert-base-cased")

>>> # Update configuration during loading
>>> model = AutoModelForNextSentencePrediction.from_pretrained("google-bert/bert-base-cased", output_attentions=True)
>>> model.config.output_attentions
True

>>> # Loading from a TF checkpoint file instead of a PyTorch model (slower)
>>> config = AutoConfig.from_pretrained("./tf_model/bert_tf_model_config.json")
>>> model = AutoModelForNextSentencePrediction.from_pretrained(
...     "./tf_model/bert_tf_checkpoint.ckpt.index", from_tf=True, config=config
... )

TFAutoModelForNextSentencePrediction

class transformers.TFAutoModelForNextSentencePrediction

< >

( *args **kwargs )

This is a generic model class that will be instantiated as one of the model classes of the library (with a next sentence prediction head) when created with the from_pretrained() class method or the from_config() class method.

This class cannot be instantiated directly using __init__() (throws an error).

from_config

< >

( **kwargs )

Parameters

  • config (PretrainedConfig) — The model class to instantiate is selected based on the configuration class:
  • attn_implementation (str, optional) — The attention implementation to use in the model (if relevant). Can be any of "eager" (manual implementation of the attention), "sdpa" (using F.scaled_dot_product_attention), or "flash_attention_2" (using Dao-AILab/flash-attention). By default, if available, SDPA will be used for torch>=2.1.1. The default is otherwise the manual "eager" implementation.

Instantiates one of the model classes of the library (with a next sentence prediction head) from a configuration.

Note: Loading a model from its configuration file does not load the model weights. It only affects the model’s configuration. Use from_pretrained() to load the model weights.

Examples:

>>> from transformers import AutoConfig, TFAutoModelForNextSentencePrediction

>>> # Download configuration from huggingface.co and cache.
>>> config = AutoConfig.from_pretrained("google-bert/bert-base-cased")
>>> model = TFAutoModelForNextSentencePrediction.from_config(config)

from_pretrained

< >

( *model_args **kwargs )

Parameters

  • pretrained_model_name_or_path (str or os.PathLike) — Can be either:

    • A string, the model id of a pretrained model hosted inside a model repo on huggingface.co.
    • A path to a directory containing model weights saved using save_pretrained(), e.g., ./my_model_directory/.
    • A path or url to a PyTorch state_dict save file (e.g, ./pt_model/pytorch_model.bin). In this case, from_pt should be set to True and a configuration object should be provided as config argument. This loading path is slower than converting the PyTorch model in a TensorFlow model using the provided conversion scripts and loading the TensorFlow model afterwards.
  • model_args (additional positional arguments, optional) — Will be passed along to the underlying model __init__() method.
  • config (PretrainedConfig, optional) — Configuration for the model to use instead of an automatically loaded configuration. Configuration can be automatically loaded when:

    • The model is a model provided by the library (loaded with the model id string of a pretrained model).
    • The model was saved using save_pretrained() and is reloaded by supplying the save directory.
    • The model is loaded by supplying a local directory as pretrained_model_name_or_path and a configuration JSON file named config.json is found in the directory.
  • cache_dir (str or os.PathLike, optional) — Path to a directory in which a downloaded pretrained model configuration should be cached if the standard cache should not be used.
  • from_pt (bool, optional, defaults to False) — Load the model weights from a PyTorch checkpoint save file (see docstring of pretrained_model_name_or_path argument).
  • force_download (bool, optional, defaults to False) — Whether or not to force the (re-)download of the model weights and configuration files, overriding the cached versions if they exist.
  • resume_download (bool, optional, defaults to False) — Whether or not to delete incompletely received files. Will attempt to resume the download if such a file exists.
  • proxies (Dict[str, str], optional) — A dictionary of proxy servers to use by protocol or endpoint, e.g., {'http': 'foo.bar:3128', 'http://hostname': 'foo.bar:4012'}. The proxies are used on each request.
  • output_loading_info(bool, optional, defaults to False) — Whether ot not to also return a dictionary containing missing keys, unexpected keys and error messages.
  • local_files_only(bool, optional, defaults to False) — Whether or not to only look at local files (e.g., not try downloading the model).
  • revision (str, optional, defaults to "main") — The specific model version to use. It can be a branch name, a tag name, or a commit id, since we use a git-based system for storing models and other artifacts on huggingface.co, so revision can be any identifier allowed by git.
  • trust_remote_code (bool, optional, defaults to False) — Whether or not to allow for custom models defined on the Hub in their own modeling files. This option should only be set to True for repositories you trust and in which you have read the code, as it will execute code present on the Hub on your local machine.
  • code_revision (str, optional, defaults to "main") — The specific revision to use for the code on the Hub, if the code leaves in a different repository than the rest of the model. It can be a branch name, a tag name, or a commit id, since we use a git-based system for storing models and other artifacts on huggingface.co, so revision can be any identifier allowed by git.
  • kwargs (additional keyword arguments, optional) — Can be used to update the configuration object (after it being loaded) and initiate the model (e.g., output_attentions=True). Behaves differently depending on whether a config is provided or automatically loaded:

    • If a configuration is provided with config, **kwargs will be directly passed to the underlying model’s __init__ method (we assume all relevant updates to the configuration have already been done)
    • If a configuration is not provided, kwargs will be first passed to the configuration class initialization function (from_pretrained()). Each key of kwargs that corresponds to a configuration attribute will be used to override said attribute with the supplied kwargs value. Remaining keys that do not correspond to any configuration attribute will be passed to the underlying model’s __init__ function.

Instantiate one of the model classes of the library (with a next sentence prediction head) from a pretrained model.

The model class to instantiate is selected based on the model_type property of the config object (either passed as an argument or loaded from pretrained_model_name_or_path if possible), or when it’s missing, by falling back to using pattern matching on pretrained_model_name_or_path:

Examples:

>>> from transformers import AutoConfig, TFAutoModelForNextSentencePrediction

>>> # Download model and configuration from huggingface.co and cache.
>>> model = TFAutoModelForNextSentencePrediction.from_pretrained("google-bert/bert-base-cased")

>>> # Update configuration during loading
>>> model = TFAutoModelForNextSentencePrediction.from_pretrained("google-bert/bert-base-cased", output_attentions=True)
>>> model.config.output_attentions
True

>>> # Loading from a PyTorch checkpoint file instead of a TensorFlow model (slower)
>>> config = AutoConfig.from_pretrained("./pt_model/bert_pt_model_config.json")
>>> model = TFAutoModelForNextSentencePrediction.from_pretrained(
...     "./pt_model/bert_pytorch_model.bin", from_pt=True, config=config
... )

FlaxAutoModelForNextSentencePrediction

class transformers.FlaxAutoModelForNextSentencePrediction

< >

( *args **kwargs )

This is a generic model class that will be instantiated as one of the model classes of the library (with a next sentence prediction head) when created with the from_pretrained() class method or the from_config() class method.

This class cannot be instantiated directly using __init__() (throws an error).

from_config

< >

( **kwargs )

Parameters

  • config (PretrainedConfig) — The model class to instantiate is selected based on the configuration class:
  • attn_implementation (str, optional) — The attention implementation to use in the model (if relevant). Can be any of "eager" (manual implementation of the attention), "sdpa" (using F.scaled_dot_product_attention), or "flash_attention_2" (using Dao-AILab/flash-attention). By default, if available, SDPA will be used for torch>=2.1.1. The default is otherwise the manual "eager" implementation.

Instantiates one of the model classes of the library (with a next sentence prediction head) from a configuration.

Note: Loading a model from its configuration file does not load the model weights. It only affects the model’s configuration. Use from_pretrained() to load the model weights.

Examples:

>>> from transformers import AutoConfig, FlaxAutoModelForNextSentencePrediction

>>> # Download configuration from huggingface.co and cache.
>>> config = AutoConfig.from_pretrained("google-bert/bert-base-cased")
>>> model = FlaxAutoModelForNextSentencePrediction.from_config(config)

from_pretrained

< >

( *model_args **kwargs )

Parameters

  • pretrained_model_name_or_path (str or os.PathLike) — Can be either:

    • A string, the model id of a pretrained model hosted inside a model repo on huggingface.co.
    • A path to a directory containing model weights saved using save_pretrained(), e.g., ./my_model_directory/.
    • A path or url to a PyTorch state_dict save file (e.g, ./pt_model/pytorch_model.bin). In this case, from_pt should be set to True and a configuration object should be provided as config argument. This loading path is slower than converting the PyTorch model in a TensorFlow model using the provided conversion scripts and loading the TensorFlow model afterwards.
  • model_args (additional positional arguments, optional) — Will be passed along to the underlying model __init__() method.
  • config (PretrainedConfig, optional) — Configuration for the model to use instead of an automatically loaded configuration. Configuration can be automatically loaded when:

    • The model is a model provided by the library (loaded with the model id string of a pretrained model).
    • The model was saved using save_pretrained() and is reloaded by supplying the save directory.
    • The model is loaded by supplying a local directory as pretrained_model_name_or_path and a configuration JSON file named config.json is found in the directory.
  • cache_dir (str or os.PathLike, optional) — Path to a directory in which a downloaded pretrained model configuration should be cached if the standard cache should not be used.
  • from_pt (bool, optional, defaults to False) — Load the model weights from a PyTorch checkpoint save file (see docstring of pretrained_model_name_or_path argument).
  • force_download (bool, optional, defaults to False) — Whether or not to force the (re-)download of the model weights and configuration files, overriding the cached versions if they exist.
  • resume_download (bool, optional, defaults to False) — Whether or not to delete incompletely received files. Will attempt to resume the download if such a file exists.
  • proxies (Dict[str, str], optional) — A dictionary of proxy servers to use by protocol or endpoint, e.g., {'http': 'foo.bar:3128', 'http://hostname': 'foo.bar:4012'}. The proxies are used on each request.
  • output_loading_info(bool, optional, defaults to False) — Whether ot not to also return a dictionary containing missing keys, unexpected keys and error messages.
  • local_files_only(bool, optional, defaults to False) — Whether or not to only look at local files (e.g., not try downloading the model).
  • revision (str, optional, defaults to "main") — The specific model version to use. It can be a branch name, a tag name, or a commit id, since we use a git-based system for storing models and other artifacts on huggingface.co, so revision can be any identifier allowed by git.
  • trust_remote_code (bool, optional, defaults to False) — Whether or not to allow for custom models defined on the Hub in their own modeling files. This option should only be set to True for repositories you trust and in which you have read the code, as it will execute code present on the Hub on your local machine.
  • code_revision (str, optional, defaults to "main") — The specific revision to use for the code on the Hub, if the code leaves in a different repository than the rest of the model. It can be a branch name, a tag name, or a commit id, since we use a git-based system for storing models and other artifacts on huggingface.co, so revision can be any identifier allowed by git.
  • kwargs (additional keyword arguments, optional) — Can be used to update the configuration object (after it being loaded) and initiate the model (e.g., output_attentions=True). Behaves differently depending on whether a config is provided or automatically loaded:

    • If a configuration is provided with config, **kwargs will be directly passed to the underlying model’s __init__ method (we assume all relevant updates to the configuration have already been done)
    • If a configuration is not provided, kwargs will be first passed to the configuration class initialization function (from_pretrained()). Each key of kwargs that corresponds to a configuration attribute will be used to override said attribute with the supplied kwargs value. Remaining keys that do not correspond to any configuration attribute will be passed to the underlying model’s __init__ function.

Instantiate one of the model classes of the library (with a next sentence prediction head) from a pretrained model.

The model class to instantiate is selected based on the model_type property of the config object (either passed as an argument or loaded from pretrained_model_name_or_path if possible), or when it’s missing, by falling back to using pattern matching on pretrained_model_name_or_path:

Examples:

>>> from transformers import AutoConfig, FlaxAutoModelForNextSentencePrediction

>>> # Download model and configuration from huggingface.co and cache.
>>> model = FlaxAutoModelForNextSentencePrediction.from_pretrained("google-bert/bert-base-cased")

>>> # Update configuration during loading
>>> model = FlaxAutoModelForNextSentencePrediction.from_pretrained("google-bert/bert-base-cased", output_attentions=True)
>>> model.config.output_attentions
True

>>> # Loading from a PyTorch checkpoint file instead of a TensorFlow model (slower)
>>> config = AutoConfig.from_pretrained("./pt_model/bert_pt_model_config.json")
>>> model = FlaxAutoModelForNextSentencePrediction.from_pretrained(
...     "./pt_model/bert_pytorch_model.bin", from_pt=True, config=config
... )

AutoModelForTokenClassification

class transformers.AutoModelForTokenClassification

< >

( *args **kwargs )

This is a generic model class that will be instantiated as one of the model classes of the library (with a token classification head) when created with the from_pretrained() class method or the from_config() class method.

This class cannot be instantiated directly using __init__() (throws an error).

from_config

< >

( **kwargs )

Parameters

  • config (PretrainedConfig) — The model class to instantiate is selected based on the configuration class:

    • AlbertConfig configuration class: AlbertForTokenClassification (ALBERT model)
    • BertConfig configuration class: BertForTokenClassification (BERT model)
    • BigBirdConfig configuration class: BigBirdForTokenClassification (BigBird model)
    • BioGptConfig configuration class: BioGptForTokenClassification (BioGpt model)
    • BloomConfig configuration class: BloomForTokenClassification (BLOOM model)
    • BrosConfig configuration class: BrosForTokenClassification (BROS model)
    • CamembertConfig configuration class: CamembertForTokenClassification (CamemBERT model)
    • CanineConfig configuration class: CanineForTokenClassification (CANINE model)
    • ConvBertConfig configuration class: ConvBertForTokenClassification (ConvBERT model)
    • Data2VecTextConfig configuration class: Data2VecTextForTokenClassification (Data2VecText model)
    • DebertaConfig configuration class: DebertaForTokenClassification (DeBERTa model)
    • DebertaV2Config configuration class: DebertaV2ForTokenClassification (DeBERTa-v2 model)
    • DistilBertConfig configuration class: DistilBertForTokenClassification (DistilBERT model)
    • ElectraConfig configuration class: ElectraForTokenClassification (ELECTRA model)
    • ErnieConfig configuration class: ErnieForTokenClassification (ERNIE model)
    • ErnieMConfig configuration class: ErnieMForTokenClassification (ErnieM model)
    • EsmConfig configuration class: EsmForTokenClassification (ESM model)
    • FNetConfig configuration class: FNetForTokenClassification (FNet model)
    • FalconConfig configuration class: FalconForTokenClassification (Falcon model)
    • FlaubertConfig configuration class: FlaubertForTokenClassification (FlauBERT model)
    • FunnelConfig configuration class: FunnelForTokenClassification (Funnel Transformer model)
    • GPT2Config configuration class: GPT2ForTokenClassification (OpenAI GPT-2 model)
    • GPTBigCodeConfig configuration class: GPTBigCodeForTokenClassification (GPTBigCode model)
    • GPTNeoConfig configuration class: GPTNeoForTokenClassification (GPT Neo model)
    • GPTNeoXConfig configuration class: GPTNeoXForTokenClassification (GPT NeoX model)
    • IBertConfig configuration class: IBertForTokenClassification (I-BERT model)
    • LayoutLMConfig configuration class: LayoutLMForTokenClassification (LayoutLM model)
    • LayoutLMv2Config configuration class: LayoutLMv2ForTokenClassification (LayoutLMv2 model)
    • LayoutLMv3Config configuration class: LayoutLMv3ForTokenClassification (LayoutLMv3 model)
    • LiltConfig configuration class: LiltForTokenClassification (LiLT model)
    • LongformerConfig configuration class: LongformerForTokenClassification (Longformer model)
    • LukeConfig configuration class: LukeForTokenClassification (LUKE model)
    • MPNetConfig configuration class: MPNetForTokenClassification (MPNet model)
    • MT5Config configuration class: MT5ForTokenClassification (MT5 model)
    • MarkupLMConfig configuration class: MarkupLMForTokenClassification (MarkupLM model)
    • MegaConfig configuration class: MegaForTokenClassification (MEGA model)
    • MegatronBertConfig configuration class: MegatronBertForTokenClassification (Megatron-BERT model)
    • MobileBertConfig configuration class: MobileBertForTokenClassification (MobileBERT model)
    • MptConfig configuration class: MptForTokenClassification (MPT model)
    • MraConfig configuration class: MraForTokenClassification (MRA model)
    • NezhaConfig configuration class: NezhaForTokenClassification (Nezha model)
    • NystromformerConfig configuration class: NystromformerForTokenClassification (Nyströmformer model)
    • PhiConfig configuration class: PhiForTokenClassification (Phi model)
    • QDQBertConfig configuration class: QDQBertForTokenClassification (QDQBert model)
    • RemBertConfig configuration class: RemBertForTokenClassification (RemBERT model)
    • RoCBertConfig configuration class: RoCBertForTokenClassification (RoCBert model)
    • RoFormerConfig configuration class: RoFormerForTokenClassification (RoFormer model)
    • RobertaConfig configuration class: RobertaForTokenClassification (RoBERTa model)
    • RobertaPreLayerNormConfig configuration class: RobertaPreLayerNormForTokenClassification (RoBERTa-PreLayerNorm model)
    • SqueezeBertConfig configuration class: SqueezeBertForTokenClassification (SqueezeBERT model)
    • T5Config configuration class: T5ForTokenClassification (T5 model)
    • UMT5Config configuration class: UMT5ForTokenClassification (UMT5 model)
    • XLMConfig configuration class: XLMForTokenClassification (XLM model)
    • XLMRobertaConfig configuration class: XLMRobertaForTokenClassification (XLM-RoBERTa model)
    • XLMRobertaXLConfig configuration class: XLMRobertaXLForTokenClassification (XLM-RoBERTa-XL model)
    • XLNetConfig configuration class: XLNetForTokenClassification (XLNet model)
    • XmodConfig configuration class: XmodForTokenClassification (X-MOD model)
    • YosoConfig configuration class: YosoForTokenClassification (YOSO model)
  • attn_implementation (str, optional) — The attention implementation to use in the model (if relevant). Can be any of "eager" (manual implementation of the attention), "sdpa" (using F.scaled_dot_product_attention), or "flash_attention_2" (using Dao-AILab/flash-attention). By default, if available, SDPA will be used for torch>=2.1.1. The default is otherwise the manual "eager" implementation.

Instantiates one of the model classes of the library (with a token classification head) from a configuration.

Note: Loading a model from its configuration file does not load the model weights. It only affects the model’s configuration. Use from_pretrained() to load the model weights.

Examples:

>>> from transformers import AutoConfig, AutoModelForTokenClassification

>>> # Download configuration from huggingface.co and cache.
>>> config = AutoConfig.from_pretrained("google-bert/bert-base-cased")
>>> model = AutoModelForTokenClassification.from_config(config)

from_pretrained

< >

( *model_args **kwargs )

Parameters

  • pretrained_model_name_or_path (str or os.PathLike) — Can be either:

    • A string, the model id of a pretrained model hosted inside a model repo on huggingface.co.
    • A path to a directory containing model weights saved using save_pretrained(), e.g., ./my_model_directory/.
    • A path or url to a tensorflow index checkpoint file (e.g, ./tf_model/model.ckpt.index). In this case, from_tf should be set to True and a configuration object should be provided as config argument. This loading path is slower than converting the TensorFlow checkpoint in a PyTorch model using the provided conversion scripts and loading the PyTorch model afterwards.
  • model_args (additional positional arguments, optional) — Will be passed along to the underlying model __init__() method.
  • config (PretrainedConfig, optional) — Configuration for the model to use instead of an automatically loaded configuration. Configuration can be automatically loaded when:

    • The model is a model provided by the library (loaded with the model id string of a pretrained model).
    • The model was saved using save_pretrained() and is reloaded by supplying the save directory.
    • The model is loaded by supplying a local directory as pretrained_model_name_or_path and a configuration JSON file named config.json is found in the directory.
  • state_dict (Dict[str, torch.Tensor], optional) — A state dictionary to use instead of a state dictionary loaded from saved weights file.

    This option can be used if you want to create a model from a pretrained configuration but load your own weights. In this case though, you should check if using save_pretrained() and from_pretrained() is not a simpler option.

  • cache_dir (str or os.PathLike, optional) — Path to a directory in which a downloaded pretrained model configuration should be cached if the standard cache should not be used.
  • from_tf (bool, optional, defaults to False) — Load the model weights from a TensorFlow checkpoint save file (see docstring of pretrained_model_name_or_path argument).
  • force_download (bool, optional, defaults to False) — Whether or not to force the (re-)download of the model weights and configuration files, overriding the cached versions if they exist.
  • resume_download (bool, optional, defaults to False) — Whether or not to delete incompletely received files. Will attempt to resume the download if such a file exists.
  • proxies (Dict[str, str], optional) — A dictionary of proxy servers to use by protocol or endpoint, e.g., {'http': 'foo.bar:3128', 'http://hostname': 'foo.bar:4012'}. The proxies are used on each request.
  • output_loading_info(bool, optional, defaults to False) — Whether ot not to also return a dictionary containing missing keys, unexpected keys and error messages.
  • local_files_only(bool, optional, defaults to False) — Whether or not to only look at local files (e.g., not try downloading the model).
  • revision (str, optional, defaults to "main") — The specific model version to use. It can be a branch name, a tag name, or a commit id, since we use a git-based system for storing models and other artifacts on huggingface.co, so revision can be any identifier allowed by git.
  • trust_remote_code (bool, optional, defaults to False) — Whether or not to allow for custom models defined on the Hub in their own modeling files. This option should only be set to True for repositories you trust and in which you have read the code, as it will execute code present on the Hub on your local machine.
  • code_revision (str, optional, defaults to "main") — The specific revision to use for the code on the Hub, if the code leaves in a different repository than the rest of the model. It can be a branch name, a tag name, or a commit id, since we use a git-based system for storing models and other artifacts on huggingface.co, so revision can be any identifier allowed by git.
  • kwargs (additional keyword arguments, optional) — Can be used to update the configuration object (after it being loaded) and initiate the model (e.g., output_attentions=True). Behaves differently depending on whether a config is provided or automatically loaded:

    • If a configuration is provided with config, **kwargs will be directly passed to the underlying model’s __init__ method (we assume all relevant updates to the configuration have already been done)
    • If a configuration is not provided, kwargs will be first passed to the configuration class initialization function (from_pretrained()). Each key of kwargs that corresponds to a configuration attribute will be used to override said attribute with the supplied kwargs value. Remaining keys that do not correspond to any configuration attribute will be passed to the underlying model’s __init__ function.

Instantiate one of the model classes of the library (with a token classification head) from a pretrained model.

The model class to instantiate is selected based on the model_type property of the config object (either passed as an argument or loaded from pretrained_model_name_or_path if possible), or when it’s missing, by falling back to using pattern matching on pretrained_model_name_or_path:

  • albert — AlbertForTokenClassification (ALBERT model)
  • bert — BertForTokenClassification (BERT model)
  • big_bird — BigBirdForTokenClassification (BigBird model)
  • biogpt — BioGptForTokenClassification (BioGpt model)
  • bloom — BloomForTokenClassification (BLOOM model)
  • bros — BrosForTokenClassification (BROS model)
  • camembert — CamembertForTokenClassification (CamemBERT model)
  • canine — CanineForTokenClassification (CANINE model)
  • convbert — ConvBertForTokenClassification (ConvBERT model)
  • data2vec-text — Data2VecTextForTokenClassification (Data2VecText model)
  • deberta — DebertaForTokenClassification (DeBERTa model)
  • deberta-v2 — DebertaV2ForTokenClassification (DeBERTa-v2 model)
  • distilbert — DistilBertForTokenClassification (DistilBERT model)
  • electra — ElectraForTokenClassification (ELECTRA model)
  • ernie — ErnieForTokenClassification (ERNIE model)
  • ernie_m — ErnieMForTokenClassification (ErnieM model)
  • esm — EsmForTokenClassification (ESM model)
  • falcon — FalconForTokenClassification (Falcon model)
  • flaubert — FlaubertForTokenClassification (FlauBERT model)
  • fnet — FNetForTokenClassification (FNet model)
  • funnel — FunnelForTokenClassification (Funnel Transformer model)
  • gpt-sw3 — GPT2ForTokenClassification (GPT-Sw3 model)
  • gpt2 — GPT2ForTokenClassification (OpenAI GPT-2 model)
  • gpt_bigcode — GPTBigCodeForTokenClassification (GPTBigCode model)
  • gpt_neo — GPTNeoForTokenClassification (GPT Neo model)
  • gpt_neox — GPTNeoXForTokenClassification (GPT NeoX model)
  • ibert — IBertForTokenClassification (I-BERT model)
  • layoutlm — LayoutLMForTokenClassification (LayoutLM model)
  • layoutlmv2 — LayoutLMv2ForTokenClassification (LayoutLMv2 model)
  • layoutlmv3 — LayoutLMv3ForTokenClassification (LayoutLMv3 model)
  • lilt — LiltForTokenClassification (LiLT model)
  • longformer — LongformerForTokenClassification (Longformer model)
  • luke — LukeForTokenClassification (LUKE model)
  • markuplm — MarkupLMForTokenClassification (MarkupLM model)
  • mega — MegaForTokenClassification (MEGA model)
  • megatron-bert — MegatronBertForTokenClassification (Megatron-BERT model)
  • mobilebert — MobileBertForTokenClassification (MobileBERT model)
  • mpnet — MPNetForTokenClassification (MPNet model)
  • mpt — MptForTokenClassification (MPT model)
  • mra — MraForTokenClassification (MRA model)
  • mt5 — MT5ForTokenClassification (MT5 model)
  • nezha — NezhaForTokenClassification (Nezha model)
  • nystromformer — NystromformerForTokenClassification (Nyströmformer model)
  • phi — PhiForTokenClassification (Phi model)
  • qdqbert — QDQBertForTokenClassification (QDQBert model)
  • rembert — RemBertForTokenClassification (RemBERT model)
  • roberta — RobertaForTokenClassification (RoBERTa model)
  • roberta-prelayernorm — RobertaPreLayerNormForTokenClassification (RoBERTa-PreLayerNorm model)
  • roc_bert — RoCBertForTokenClassification (RoCBert model)
  • roformer — RoFormerForTokenClassification (RoFormer model)
  • squeezebert — SqueezeBertForTokenClassification (SqueezeBERT model)
  • t5 — T5ForTokenClassification (T5 model)
  • umt5 — UMT5ForTokenClassification (UMT5 model)
  • xlm — XLMForTokenClassification (XLM model)
  • xlm-roberta — XLMRobertaForTokenClassification (XLM-RoBERTa model)
  • xlm-roberta-xl — XLMRobertaXLForTokenClassification (XLM-RoBERTa-XL model)
  • xlnet — XLNetForTokenClassification (XLNet model)
  • xmod — XmodForTokenClassification (X-MOD model)
  • yoso — YosoForTokenClassification (YOSO model)

The model is set in evaluation mode by default using model.eval() (so for instance, dropout modules are deactivated). To train the model, you should first set it back in training mode with model.train()

Examples:

>>> from transformers import AutoConfig, AutoModelForTokenClassification

>>> # Download model and configuration from huggingface.co and cache.
>>> model = AutoModelForTokenClassification.from_pretrained("google-bert/bert-base-cased")

>>> # Update configuration during loading
>>> model = AutoModelForTokenClassification.from_pretrained("google-bert/bert-base-cased", output_attentions=True)
>>> model.config.output_attentions
True

>>> # Loading from a TF checkpoint file instead of a PyTorch model (slower)
>>> config = AutoConfig.from_pretrained("./tf_model/bert_tf_model_config.json")
>>> model = AutoModelForTokenClassification.from_pretrained(
...     "./tf_model/bert_tf_checkpoint.ckpt.index", from_tf=True, config=config
... )

TFAutoModelForTokenClassification

class transformers.TFAutoModelForTokenClassification

< >

( *args **kwargs )

This is a generic model class that will be instantiated as one of the model classes of the library (with a token classification head) when created with the from_pretrained() class method or the from_config() class method.

This class cannot be instantiated directly using __init__() (throws an error).

from_config

< >

( **kwargs )

Parameters

  • config (PretrainedConfig) — The model class to instantiate is selected based on the configuration class:

    • AlbertConfig configuration class: TFAlbertForTokenClassification (ALBERT model)
    • BertConfig configuration class: TFBertForTokenClassification (BERT model)
    • CamembertConfig configuration class: TFCamembertForTokenClassification (CamemBERT model)
    • ConvBertConfig configuration class: TFConvBertForTokenClassification (ConvBERT model)
    • DebertaConfig configuration class: TFDebertaForTokenClassification (DeBERTa model)
    • DebertaV2Config configuration class: TFDebertaV2ForTokenClassification (DeBERTa-v2 model)
    • DistilBertConfig configuration class: TFDistilBertForTokenClassification (DistilBERT model)
    • ElectraConfig configuration class: TFElectraForTokenClassification (ELECTRA model)
    • EsmConfig configuration class: TFEsmForTokenClassification (ESM model)
    • FlaubertConfig configuration class: TFFlaubertForTokenClassification (FlauBERT model)
    • FunnelConfig configuration class: TFFunnelForTokenClassification (Funnel Transformer model)
    • LayoutLMConfig configuration class: TFLayoutLMForTokenClassification (LayoutLM model)
    • LayoutLMv3Config configuration class: TFLayoutLMv3ForTokenClassification (LayoutLMv3 model)
    • LongformerConfig configuration class: TFLongformerForTokenClassification (Longformer model)
    • MPNetConfig configuration class: TFMPNetForTokenClassification (MPNet model)
    • MobileBertConfig configuration class: TFMobileBertForTokenClassification (MobileBERT model)
    • RemBertConfig configuration class: TFRemBertForTokenClassification (RemBERT model)
    • RoFormerConfig configuration class: TFRoFormerForTokenClassification (RoFormer model)
    • RobertaConfig configuration class: TFRobertaForTokenClassification (RoBERTa model)
    • RobertaPreLayerNormConfig configuration class: TFRobertaPreLayerNormForTokenClassification (RoBERTa-PreLayerNorm model)
    • XLMConfig configuration class: TFXLMForTokenClassification (XLM model)
    • XLMRobertaConfig configuration class: TFXLMRobertaForTokenClassification (XLM-RoBERTa model)
    • XLNetConfig configuration class: TFXLNetForTokenClassification (XLNet model)
  • attn_implementation (str, optional) — The attention implementation to use in the model (if relevant). Can be any of "eager" (manual implementation of the attention), "sdpa" (using F.scaled_dot_product_attention), or "flash_attention_2" (using Dao-AILab/flash-attention). By default, if available, SDPA will be used for torch>=2.1.1. The default is otherwise the manual "eager" implementation.

Instantiates one of the model classes of the library (with a token classification head) from a configuration.

Note: Loading a model from its configuration file does not load the model weights. It only affects the model’s configuration. Use from_pretrained() to load the model weights.

Examples:

>>> from transformers import AutoConfig, TFAutoModelForTokenClassification

>>> # Download configuration from huggingface.co and cache.
>>> config = AutoConfig.from_pretrained("google-bert/bert-base-cased")
>>> model = TFAutoModelForTokenClassification.from_config(config)

from_pretrained

< >

( *model_args **kwargs )

Parameters

  • pretrained_model_name_or_path (str or os.PathLike) — Can be either:

    • A string, the model id of a pretrained model hosted inside a model repo on huggingface.co.
    • A path to a directory containing model weights saved using save_pretrained(), e.g., ./my_model_directory/.
    • A path or url to a PyTorch state_dict save file (e.g, ./pt_model/pytorch_model.bin). In this case, from_pt should be set to True and a configuration object should be provided as config argument. This loading path is slower than converting the PyTorch model in a TensorFlow model using the provided conversion scripts and loading the TensorFlow model afterwards.
  • model_args (additional positional arguments, optional) — Will be passed along to the underlying model __init__() method.
  • config (PretrainedConfig, optional) — Configuration for the model to use instead of an automatically loaded configuration. Configuration can be automatically loaded when:

    • The model is a model provided by the library (loaded with the model id string of a pretrained model).
    • The model was saved using save_pretrained() and is reloaded by supplying the save directory.
    • The model is loaded by supplying a local directory as pretrained_model_name_or_path and a configuration JSON file named config.json is found in the directory.
  • cache_dir (str or os.PathLike, optional) — Path to a directory in which a downloaded pretrained model configuration should be cached if the standard cache should not be used.
  • from_pt (bool, optional, defaults to False) — Load the model weights from a PyTorch checkpoint save file (see docstring of pretrained_model_name_or_path argument).
  • force_download (bool, optional, defaults to False) — Whether or not to force the (re-)download of the model weights and configuration files, overriding the cached versions if they exist.
  • resume_download (bool, optional, defaults to False) — Whether or not to delete incompletely received files. Will attempt to resume the download if such a file exists.
  • proxies (Dict[str, str], optional) — A dictionary of proxy servers to use by protocol or endpoint, e.g., {'http': 'foo.bar:3128', 'http://hostname': 'foo.bar:4012'}. The proxies are used on each request.
  • output_loading_info(bool, optional, defaults to False) — Whether ot not to also return a dictionary containing missing keys, unexpected keys and error messages.
  • local_files_only(bool, optional, defaults to False) — Whether or not to only look at local files (e.g., not try downloading the model).
  • revision (str, optional, defaults to "main") — The specific model version to use. It can be a branch name, a tag name, or a commit id, since we use a git-based system for storing models and other artifacts on huggingface.co, so revision can be any identifier allowed by git.
  • trust_remote_code (bool, optional, defaults to False) — Whether or not to allow for custom models defined on the Hub in their own modeling files. This option should only be set to True for repositories you trust and in which you have read the code, as it will execute code present on the Hub on your local machine.
  • code_revision (str, optional, defaults to "main") — The specific revision to use for the code on the Hub, if the code leaves in a different repository than the rest of the model. It can be a branch name, a tag name, or a commit id, since we use a git-based system for storing models and other artifacts on huggingface.co, so revision can be any identifier allowed by git.
  • kwargs (additional keyword arguments, optional) — Can be used to update the configuration object (after it being loaded) and initiate the model (e.g., output_attentions=True). Behaves differently depending on whether a config is provided or automatically loaded:

    • If a configuration is provided with config, **kwargs will be directly passed to the underlying model’s __init__ method (we assume all relevant updates to the configuration have already been done)
    • If a configuration is not provided, kwargs will be first passed to the configuration class initialization function (from_pretrained()). Each key of kwargs that corresponds to a configuration attribute will be used to override said attribute with the supplied kwargs value. Remaining keys that do not correspond to any configuration attribute will be passed to the underlying model’s __init__ function.

Instantiate one of the model classes of the library (with a token classification head) from a pretrained model.

The model class to instantiate is selected based on the model_type property of the config object (either passed as an argument or loaded from pretrained_model_name_or_path if possible), or when it’s missing, by falling back to using pattern matching on pretrained_model_name_or_path:

  • albert — TFAlbertForTokenClassification (ALBERT model)
  • bert — TFBertForTokenClassification (BERT model)
  • camembert — TFCamembertForTokenClassification (CamemBERT model)
  • convbert — TFConvBertForTokenClassification (ConvBERT model)
  • deberta — TFDebertaForTokenClassification (DeBERTa model)
  • deberta-v2 — TFDebertaV2ForTokenClassification (DeBERTa-v2 model)
  • distilbert — TFDistilBertForTokenClassification (DistilBERT model)
  • electra — TFElectraForTokenClassification (ELECTRA model)
  • esm — TFEsmForTokenClassification (ESM model)
  • flaubert — TFFlaubertForTokenClassification (FlauBERT model)
  • funnel — TFFunnelForTokenClassification (Funnel Transformer model)
  • layoutlm — TFLayoutLMForTokenClassification (LayoutLM model)
  • layoutlmv3 — TFLayoutLMv3ForTokenClassification (LayoutLMv3 model)
  • longformer — TFLongformerForTokenClassification (Longformer model)
  • mobilebert — TFMobileBertForTokenClassification (MobileBERT model)
  • mpnet — TFMPNetForTokenClassification (MPNet model)
  • rembert — TFRemBertForTokenClassification (RemBERT model)
  • roberta — TFRobertaForTokenClassification (RoBERTa model)
  • roberta-prelayernorm — TFRobertaPreLayerNormForTokenClassification (RoBERTa-PreLayerNorm model)
  • roformer — TFRoFormerForTokenClassification (RoFormer model)
  • xlm — TFXLMForTokenClassification (XLM model)
  • xlm-roberta — TFXLMRobertaForTokenClassification (XLM-RoBERTa model)
  • xlnet — TFXLNetForTokenClassification (XLNet model)

Examples:

>>> from transformers import AutoConfig, TFAutoModelForTokenClassification

>>> # Download model and configuration from huggingface.co and cache.
>>> model = TFAutoModelForTokenClassification.from_pretrained("google-bert/bert-base-cased")

>>> # Update configuration during loading
>>> model = TFAutoModelForTokenClassification.from_pretrained("google-bert/bert-base-cased", output_attentions=True)
>>> model.config.output_attentions
True

>>> # Loading from a PyTorch checkpoint file instead of a TensorFlow model (slower)
>>> config = AutoConfig.from_pretrained("./pt_model/bert_pt_model_config.json")
>>> model = TFAutoModelForTokenClassification.from_pretrained(
...     "./pt_model/bert_pytorch_model.bin", from_pt=True, config=config
... )

FlaxAutoModelForTokenClassification

class transformers.FlaxAutoModelForTokenClassification

< >

( *args **kwargs )

This is a generic model class that will be instantiated as one of the model classes of the library (with a token classification head) when created with the from_pretrained() class method or the from_config() class method.

This class cannot be instantiated directly using __init__() (throws an error).

from_config

< >

( **kwargs )

Parameters

  • config (PretrainedConfig) — The model class to instantiate is selected based on the configuration class:

    • AlbertConfig configuration class: FlaxAlbertForTokenClassification (ALBERT model)
    • BertConfig configuration class: FlaxBertForTokenClassification (BERT model)
    • BigBirdConfig configuration class: FlaxBigBirdForTokenClassification (BigBird model)
    • DistilBertConfig configuration class: FlaxDistilBertForTokenClassification (DistilBERT model)
    • ElectraConfig configuration class: FlaxElectraForTokenClassification (ELECTRA model)
    • RoFormerConfig configuration class: FlaxRoFormerForTokenClassification (RoFormer model)
    • RobertaConfig configuration class: FlaxRobertaForTokenClassification (RoBERTa model)
    • RobertaPreLayerNormConfig configuration class: FlaxRobertaPreLayerNormForTokenClassification (RoBERTa-PreLayerNorm model)
    • XLMRobertaConfig configuration class: FlaxXLMRobertaForTokenClassification (XLM-RoBERTa model)
  • attn_implementation (str, optional) — The attention implementation to use in the model (if relevant). Can be any of "eager" (manual implementation of the attention), "sdpa" (using F.scaled_dot_product_attention), or "flash_attention_2" (using Dao-AILab/flash-attention). By default, if available, SDPA will be used for torch>=2.1.1. The default is otherwise the manual "eager" implementation.

Instantiates one of the model classes of the library (with a token classification head) from a configuration.

Note: Loading a model from its configuration file does not load the model weights. It only affects the model’s configuration. Use from_pretrained() to load the model weights.

Examples:

>>> from transformers import AutoConfig, FlaxAutoModelForTokenClassification

>>> # Download configuration from huggingface.co and cache.
>>> config = AutoConfig.from_pretrained("google-bert/bert-base-cased")
>>> model = FlaxAutoModelForTokenClassification.from_config(config)

from_pretrained

< >

( *model_args **kwargs )

Parameters

  • pretrained_model_name_or_path (str or os.PathLike) — Can be either:

    • A string, the model id of a pretrained model hosted inside a model repo on huggingface.co.
    • A path to a directory containing model weights saved using save_pretrained(), e.g., ./my_model_directory/.
    • A path or url to a PyTorch state_dict save file (e.g, ./pt_model/pytorch_model.bin). In this case, from_pt should be set to True and a configuration object should be provided as config argument. This loading path is slower than converting the PyTorch model in a TensorFlow model using the provided conversion scripts and loading the TensorFlow model afterwards.
  • model_args (additional positional arguments, optional) — Will be passed along to the underlying model __init__() method.
  • config (PretrainedConfig, optional) — Configuration for the model to use instead of an automatically loaded configuration. Configuration can be automatically loaded when:

    • The model is a model provided by the library (loaded with the model id string of a pretrained model).
    • The model was saved using save_pretrained() and is reloaded by supplying the save directory.
    • The model is loaded by supplying a local directory as pretrained_model_name_or_path and a configuration JSON file named config.json is found in the directory.
  • cache_dir (str or os.PathLike, optional) — Path to a directory in which a downloaded pretrained model configuration should be cached if the standard cache should not be used.
  • from_pt (bool, optional, defaults to False) — Load the model weights from a PyTorch checkpoint save file (see docstring of pretrained_model_name_or_path argument).
  • force_download (bool, optional, defaults to False) — Whether or not to force the (re-)download of the model weights and configuration files, overriding the cached versions if they exist.
  • resume_download (bool, optional, defaults to False) — Whether or not to delete incompletely received files. Will attempt to resume the download if such a file exists.
  • proxies (Dict[str, str], optional) — A dictionary of proxy servers to use by protocol or endpoint, e.g., {'http': 'foo.bar:3128', 'http://hostname': 'foo.bar:4012'}. The proxies are used on each request.
  • output_loading_info(bool, optional, defaults to False) — Whether ot not to also return a dictionary containing missing keys, unexpected keys and error messages.
  • local_files_only(bool, optional, defaults to False) — Whether or not to only look at local files (e.g., not try downloading the model).
  • revision (str, optional, defaults to "main") — The specific model version to use. It can be a branch name, a tag name, or a commit id, since we use a git-based system for storing models and other artifacts on huggingface.co, so revision can be any identifier allowed by git.
  • trust_remote_code (bool, optional, defaults to False) — Whether or not to allow for custom models defined on the Hub in their own modeling files. This option should only be set to True for repositories you trust and in which you have read the code, as it will execute code present on the Hub on your local machine.
  • code_revision (str, optional, defaults to "main") — The specific revision to use for the code on the Hub, if the code leaves in a different repository than the rest of the model. It can be a branch name, a tag name, or a commit id, since we use a git-based system for storing models and other artifacts on huggingface.co, so revision can be any identifier allowed by git.
  • kwargs (additional keyword arguments, optional) — Can be used to update the configuration object (after it being loaded) and initiate the model (e.g., output_attentions=True). Behaves differently depending on whether a config is provided or automatically loaded:

    • If a configuration is provided with config, **kwargs will be directly passed to the underlying model’s __init__ method (we assume all relevant updates to the configuration have already been done)
    • If a configuration is not provided, kwargs will be first passed to the configuration class initialization function (from_pretrained()). Each key of kwargs that corresponds to a configuration attribute will be used to override said attribute with the supplied kwargs value. Remaining keys that do not correspond to any configuration attribute will be passed to the underlying model’s __init__ function.

Instantiate one of the model classes of the library (with a token classification head) from a pretrained model.

The model class to instantiate is selected based on the model_type property of the config object (either passed as an argument or loaded from pretrained_model_name_or_path if possible), or when it’s missing, by falling back to using pattern matching on pretrained_model_name_or_path:

  • albert — FlaxAlbertForTokenClassification (ALBERT model)
  • bert — FlaxBertForTokenClassification (BERT model)
  • big_bird — FlaxBigBirdForTokenClassification (BigBird model)
  • distilbert — FlaxDistilBertForTokenClassification (DistilBERT model)
  • electra — FlaxElectraForTokenClassification (ELECTRA model)
  • roberta — FlaxRobertaForTokenClassification (RoBERTa model)
  • roberta-prelayernorm — FlaxRobertaPreLayerNormForTokenClassification (RoBERTa-PreLayerNorm model)
  • roformer — FlaxRoFormerForTokenClassification (RoFormer model)
  • xlm-roberta — FlaxXLMRobertaForTokenClassification (XLM-RoBERTa model)

Examples:

>>> from transformers import AutoConfig, FlaxAutoModelForTokenClassification

>>> # Download model and configuration from huggingface.co and cache.
>>> model = FlaxAutoModelForTokenClassification.from_pretrained("google-bert/bert-base-cased")

>>> # Update configuration during loading
>>> model = FlaxAutoModelForTokenClassification.from_pretrained("google-bert/bert-base-cased", output_attentions=True)
>>> model.config.output_attentions
True

>>> # Loading from a PyTorch checkpoint file instead of a TensorFlow model (slower)
>>> config = AutoConfig.from_pretrained("./pt_model/bert_pt_model_config.json")
>>> model = FlaxAutoModelForTokenClassification.from_pretrained(
...     "./pt_model/bert_pytorch_model.bin", from_pt=True, config=config
... )

AutoModelForQuestionAnswering

class transformers.AutoModelForQuestionAnswering

< >

( *args **kwargs )

This is a generic model class that will be instantiated as one of the model classes of the library (with a question answering head) when created with the from_pretrained() class method or the from_config() class method.

This class cannot be instantiated directly using __init__() (throws an error).

from_config

< >

( **kwargs )

Parameters

  • config (PretrainedConfig) — The model class to instantiate is selected based on the configuration class:

    • AlbertConfig configuration class: AlbertForQuestionAnswering (ALBERT model)
    • BartConfig configuration class: BartForQuestionAnswering (BART model)
    • BertConfig configuration class: BertForQuestionAnswering (BERT model)
    • BigBirdConfig configuration class: BigBirdForQuestionAnswering (BigBird model)
    • BigBirdPegasusConfig configuration class: BigBirdPegasusForQuestionAnswering (BigBird-Pegasus model)
    • BloomConfig configuration class: BloomForQuestionAnswering (BLOOM model)
    • CamembertConfig configuration class: CamembertForQuestionAnswering (CamemBERT model)
    • CanineConfig configuration class: CanineForQuestionAnswering (CANINE model)
    • ConvBertConfig configuration class: ConvBertForQuestionAnswering (ConvBERT model)
    • Data2VecTextConfig configuration class: Data2VecTextForQuestionAnswering (Data2VecText model)
    • DebertaConfig configuration class: DebertaForQuestionAnswering (DeBERTa model)
    • DebertaV2Config configuration class: DebertaV2ForQuestionAnswering (DeBERTa-v2 model)
    • DistilBertConfig configuration class: DistilBertForQuestionAnswering (DistilBERT model)
    • ElectraConfig configuration class: ElectraForQuestionAnswering (ELECTRA model)
    • ErnieConfig configuration class: ErnieForQuestionAnswering (ERNIE model)
    • ErnieMConfig configuration class: ErnieMForQuestionAnswering (ErnieM model)
    • FNetConfig configuration class: FNetForQuestionAnswering (FNet model)
    • FalconConfig configuration class: FalconForQuestionAnswering (Falcon model)
    • FlaubertConfig configuration class: FlaubertForQuestionAnsweringSimple (FlauBERT model)
    • FunnelConfig configuration class: FunnelForQuestionAnswering (Funnel Transformer model)
    • GPT2Config configuration class: GPT2ForQuestionAnswering (OpenAI GPT-2 model)
    • GPTJConfig configuration class: GPTJForQuestionAnswering (GPT-J model)
    • GPTNeoConfig configuration class: GPTNeoForQuestionAnswering (GPT Neo model)
    • GPTNeoXConfig configuration class: GPTNeoXForQuestionAnswering (GPT NeoX model)
    • IBertConfig configuration class: IBertForQuestionAnswering (I-BERT model)
    • LEDConfig configuration class: LEDForQuestionAnswering (LED model)
    • LayoutLMv2Config configuration class: LayoutLMv2ForQuestionAnswering (LayoutLMv2 model)
    • LayoutLMv3Config configuration class: LayoutLMv3ForQuestionAnswering (LayoutLMv3 model)
    • LiltConfig configuration class: LiltForQuestionAnswering (LiLT model)
    • LlamaConfig configuration class: LlamaForQuestionAnswering (LLaMA model)
    • LongformerConfig configuration class: LongformerForQuestionAnswering (Longformer model)
    • LukeConfig configuration class: LukeForQuestionAnswering (LUKE model)
    • LxmertConfig configuration class: LxmertForQuestionAnswering (LXMERT model)
    • MBartConfig configuration class: MBartForQuestionAnswering (mBART model)
    • MPNetConfig configuration class: MPNetForQuestionAnswering (MPNet model)
    • MT5Config configuration class: MT5ForQuestionAnswering (MT5 model)
    • MarkupLMConfig configuration class: MarkupLMForQuestionAnswering (MarkupLM model)
    • MegaConfig configuration class: MegaForQuestionAnswering (MEGA model)
    • MegatronBertConfig configuration class: MegatronBertForQuestionAnswering (Megatron-BERT model)
    • MobileBertConfig configuration class: MobileBertForQuestionAnswering (MobileBERT model)
    • MptConfig configuration class: MptForQuestionAnswering (MPT model)
    • MraConfig configuration class: MraForQuestionAnswering (MRA model)
    • MvpConfig configuration class: MvpForQuestionAnswering (MVP model)
    • NezhaConfig configuration class: NezhaForQuestionAnswering (Nezha model)
    • NystromformerConfig configuration class: NystromformerForQuestionAnswering (Nyströmformer model)
    • OPTConfig configuration class: OPTForQuestionAnswering (OPT model)
    • QDQBertConfig configuration class: QDQBertForQuestionAnswering (QDQBert model)
    • ReformerConfig configuration class: ReformerForQuestionAnswering (Reformer model)
    • RemBertConfig configuration class: RemBertForQuestionAnswering (RemBERT model)
    • RoCBertConfig configuration class: RoCBertForQuestionAnswering (RoCBert model)
    • RoFormerConfig configuration class: RoFormerForQuestionAnswering (RoFormer model)
    • RobertaConfig configuration class: RobertaForQuestionAnswering (RoBERTa model)
    • RobertaPreLayerNormConfig configuration class: RobertaPreLayerNormForQuestionAnswering (RoBERTa-PreLayerNorm model)
    • SplinterConfig configuration class: SplinterForQuestionAnswering (Splinter model)
    • SqueezeBertConfig configuration class: SqueezeBertForQuestionAnswering (SqueezeBERT model)
    • T5Config configuration class: T5ForQuestionAnswering (T5 model)
    • UMT5Config configuration class: UMT5ForQuestionAnswering (UMT5 model)
    • XLMConfig configuration class: XLMForQuestionAnsweringSimple (XLM model)
    • XLMRobertaConfig configuration class: XLMRobertaForQuestionAnswering (XLM-RoBERTa model)
    • XLMRobertaXLConfig configuration class: XLMRobertaXLForQuestionAnswering (XLM-RoBERTa-XL model)
    • XLNetConfig configuration class: XLNetForQuestionAnsweringSimple (XLNet model)
    • XmodConfig configuration class: XmodForQuestionAnswering (X-MOD model)
    • YosoConfig configuration class: YosoForQuestionAnswering (YOSO model)
  • attn_implementation (str, optional) — The attention implementation to use in the model (if relevant). Can be any of "eager" (manual implementation of the attention), "sdpa" (using F.scaled_dot_product_attention), or "flash_attention_2" (using Dao-AILab/flash-attention). By default, if available, SDPA will be used for torch>=2.1.1. The default is otherwise the manual "eager" implementation.

Instantiates one of the model classes of the library (with a question answering head) from a configuration.

Note: Loading a model from its configuration file does not load the model weights. It only affects the model’s configuration. Use from_pretrained() to load the model weights.

Examples:

>>> from transformers import AutoConfig, AutoModelForQuestionAnswering

>>> # Download configuration from huggingface.co and cache.
>>> config = AutoConfig.from_pretrained("google-bert/bert-base-cased")
>>> model = AutoModelForQuestionAnswering.from_config(config)

from_pretrained

< >

( *model_args **kwargs )

Parameters

  • pretrained_model_name_or_path (str or os.PathLike) — Can be either:

    • A string, the model id of a pretrained model hosted inside a model repo on huggingface.co.
    • A path to a directory containing model weights saved using save_pretrained(), e.g., ./my_model_directory/.
    • A path or url to a tensorflow index checkpoint file (e.g, ./tf_model/model.ckpt.index). In this case, from_tf should be set to True and a configuration object should be provided as config argument. This loading path is slower than converting the TensorFlow checkpoint in a PyTorch model using the provided conversion scripts and loading the PyTorch model afterwards.
  • model_args (additional positional arguments, optional) — Will be passed along to the underlying model __init__() method.
  • config (PretrainedConfig, optional) — Configuration for the model to use instead of an automatically loaded configuration. Configuration can be automatically loaded when:

    • The model is a model provided by the library (loaded with the model id string of a pretrained model).
    • The model was saved using save_pretrained() and is reloaded by supplying the save directory.
    • The model is loaded by supplying a local directory as pretrained_model_name_or_path and a configuration JSON file named config.json is found in the directory.
  • state_dict (Dict[str, torch.Tensor], optional) — A state dictionary to use instead of a state dictionary loaded from saved weights file.

    This option can be used if you want to create a model from a pretrained configuration but load your own weights. In this case though, you should check if using save_pretrained() and from_pretrained() is not a simpler option.

  • cache_dir (str or os.PathLike, optional) — Path to a directory in which a downloaded pretrained model configuration should be cached if the standard cache should not be used.
  • from_tf (bool, optional, defaults to False) — Load the model weights from a TensorFlow checkpoint save file (see docstring of pretrained_model_name_or_path argument).
  • force_download (bool, optional, defaults to False) — Whether or not to force the (re-)download of the model weights and configuration files, overriding the cached versions if they exist.
  • resume_download (bool, optional, defaults to False) — Whether or not to delete incompletely received files. Will attempt to resume the download if such a file exists.
  • proxies (Dict[str, str], optional) — A dictionary of proxy servers to use by protocol or endpoint, e.g., {'http': 'foo.bar:3128', 'http://hostname': 'foo.bar:4012'}. The proxies are used on each request.
  • output_loading_info(bool, optional, defaults to False) — Whether ot not to also return a dictionary containing missing keys, unexpected keys and error messages.
  • local_files_only(bool, optional, defaults to False) — Whether or not to only look at local files (e.g., not try downloading the model).
  • revision (str, optional, defaults to "main") — The specific model version to use. It can be a branch name, a tag name, or a commit id, since we use a git-based system for storing models and other artifacts on huggingface.co, so revision can be any identifier allowed by git.
  • trust_remote_code (bool, optional, defaults to False) — Whether or not to allow for custom models defined on the Hub in their own modeling files. This option should only be set to True for repositories you trust and in which you have read the code, as it will execute code present on the Hub on your local machine.
  • code_revision (str, optional, defaults to "main") — The specific revision to use for the code on the Hub, if the code leaves in a different repository than the rest of the model. It can be a branch name, a tag name, or a commit id, since we use a git-based system for storing models and other artifacts on huggingface.co, so revision can be any identifier allowed by git.
  • kwargs (additional keyword arguments, optional) — Can be used to update the configuration object (after it being loaded) and initiate the model (e.g., output_attentions=True). Behaves differently depending on whether a config is provided or automatically loaded:

    • If a configuration is provided with config, **kwargs will be directly passed to the underlying model’s __init__ method (we assume all relevant updates to the configuration have already been done)
    • If a configuration is not provided, kwargs will be first passed to the configuration class initialization function (from_pretrained()). Each key of kwargs that corresponds to a configuration attribute will be used to override said attribute with the supplied kwargs value. Remaining keys that do not correspond to any configuration attribute will be passed to the underlying model’s __init__ function.

Instantiate one of the model classes of the library (with a question answering head) from a pretrained model.

The model class to instantiate is selected based on the model_type property of the config object (either passed as an argument or loaded from pretrained_model_name_or_path if possible), or when it’s missing, by falling back to using pattern matching on pretrained_model_name_or_path:

  • albert — AlbertForQuestionAnswering (ALBERT model)
  • bart — BartForQuestionAnswering (BART model)
  • bert — BertForQuestionAnswering (BERT model)
  • big_bird — BigBirdForQuestionAnswering (BigBird model)
  • bigbird_pegasus — BigBirdPegasusForQuestionAnswering (BigBird-Pegasus model)
  • bloom — BloomForQuestionAnswering (BLOOM model)
  • camembert — CamembertForQuestionAnswering (CamemBERT model)
  • canine — CanineForQuestionAnswering (CANINE model)
  • convbert — ConvBertForQuestionAnswering (ConvBERT model)
  • data2vec-text — Data2VecTextForQuestionAnswering (Data2VecText model)
  • deberta — DebertaForQuestionAnswering (DeBERTa model)
  • deberta-v2 — DebertaV2ForQuestionAnswering (DeBERTa-v2 model)
  • distilbert — DistilBertForQuestionAnswering (DistilBERT model)
  • electra — ElectraForQuestionAnswering (ELECTRA model)
  • ernie — ErnieForQuestionAnswering (ERNIE model)
  • ernie_m — ErnieMForQuestionAnswering (ErnieM model)
  • falcon — FalconForQuestionAnswering (Falcon model)
  • flaubert — FlaubertForQuestionAnsweringSimple (FlauBERT model)
  • fnet — FNetForQuestionAnswering (FNet model)
  • funnel — FunnelForQuestionAnswering (Funnel Transformer model)
  • gpt2 — GPT2ForQuestionAnswering (OpenAI GPT-2 model)
  • gpt_neo — GPTNeoForQuestionAnswering (GPT Neo model)
  • gpt_neox — GPTNeoXForQuestionAnswering (GPT NeoX model)
  • gptj — GPTJForQuestionAnswering (GPT-J model)
  • ibert — IBertForQuestionAnswering (I-BERT model)
  • layoutlmv2 — LayoutLMv2ForQuestionAnswering (LayoutLMv2 model)
  • layoutlmv3 — LayoutLMv3ForQuestionAnswering (LayoutLMv3 model)
  • led — LEDForQuestionAnswering (LED model)
  • lilt — LiltForQuestionAnswering (LiLT model)
  • llama — LlamaForQuestionAnswering (LLaMA model)
  • longformer — LongformerForQuestionAnswering (Longformer model)
  • luke — LukeForQuestionAnswering (LUKE model)
  • lxmert — LxmertForQuestionAnswering (LXMERT model)
  • markuplm — MarkupLMForQuestionAnswering (MarkupLM model)
  • mbart — MBartForQuestionAnswering (mBART model)
  • mega — MegaForQuestionAnswering (MEGA model)
  • megatron-bert — MegatronBertForQuestionAnswering (Megatron-BERT model)
  • mobilebert — MobileBertForQuestionAnswering (MobileBERT model)
  • mpnet — MPNetForQuestionAnswering (MPNet model)
  • mpt — MptForQuestionAnswering (MPT model)
  • mra — MraForQuestionAnswering (MRA model)
  • mt5 — MT5ForQuestionAnswering (MT5 model)
  • mvp — MvpForQuestionAnswering (MVP model)
  • nezha — NezhaForQuestionAnswering (Nezha model)
  • nystromformer — NystromformerForQuestionAnswering (Nyströmformer model)
  • opt — OPTForQuestionAnswering (OPT model)
  • qdqbert — QDQBertForQuestionAnswering (QDQBert model)
  • reformer — ReformerForQuestionAnswering (Reformer model)
  • rembert — RemBertForQuestionAnswering (RemBERT model)
  • roberta — RobertaForQuestionAnswering (RoBERTa model)
  • roberta-prelayernorm — RobertaPreLayerNormForQuestionAnswering (RoBERTa-PreLayerNorm model)
  • roc_bert — RoCBertForQuestionAnswering (RoCBert model)
  • roformer — RoFormerForQuestionAnswering (RoFormer model)
  • splinter — SplinterForQuestionAnswering (Splinter model)
  • squeezebert — SqueezeBertForQuestionAnswering (SqueezeBERT model)
  • t5 — T5ForQuestionAnswering (T5 model)
  • umt5 — UMT5ForQuestionAnswering (UMT5 model)
  • xlm — XLMForQuestionAnsweringSimple (XLM model)
  • xlm-roberta — XLMRobertaForQuestionAnswering (XLM-RoBERTa model)
  • xlm-roberta-xl — XLMRobertaXLForQuestionAnswering (XLM-RoBERTa-XL model)
  • xlnet — XLNetForQuestionAnsweringSimple (XLNet model)
  • xmod — XmodForQuestionAnswering (X-MOD model)
  • yoso — YosoForQuestionAnswering (YOSO model)

The model is set in evaluation mode by default using model.eval() (so for instance, dropout modules are deactivated). To train the model, you should first set it back in training mode with model.train()

Examples:

>>> from transformers import AutoConfig, AutoModelForQuestionAnswering

>>> # Download model and configuration from huggingface.co and cache.
>>> model = AutoModelForQuestionAnswering.from_pretrained("google-bert/bert-base-cased")

>>> # Update configuration during loading
>>> model = AutoModelForQuestionAnswering.from_pretrained("google-bert/bert-base-cased", output_attentions=True)
>>> model.config.output_attentions
True

>>> # Loading from a TF checkpoint file instead of a PyTorch model (slower)
>>> config = AutoConfig.from_pretrained("./tf_model/bert_tf_model_config.json")
>>> model = AutoModelForQuestionAnswering.from_pretrained(
...     "./tf_model/bert_tf_checkpoint.ckpt.index", from_tf=True, config=config
... )

TFAutoModelForQuestionAnswering

class transformers.TFAutoModelForQuestionAnswering

< >

( *args **kwargs )

This is a generic model class that will be instantiated as one of the model classes of the library (with a question answering head) when created with the from_pretrained() class method or the from_config() class method.

This class cannot be instantiated directly using __init__() (throws an error).

from_config

< >

( **kwargs )

Parameters

  • config (PretrainedConfig) — The model class to instantiate is selected based on the configuration class:

    • AlbertConfig configuration class: TFAlbertForQuestionAnswering (ALBERT model)
    • BertConfig configuration class: TFBertForQuestionAnswering (BERT model)
    • CamembertConfig configuration class: TFCamembertForQuestionAnswering (CamemBERT model)
    • ConvBertConfig configuration class: TFConvBertForQuestionAnswering (ConvBERT model)
    • DebertaConfig configuration class: TFDebertaForQuestionAnswering (DeBERTa model)
    • DebertaV2Config configuration class: TFDebertaV2ForQuestionAnswering (DeBERTa-v2 model)
    • DistilBertConfig configuration class: TFDistilBertForQuestionAnswering (DistilBERT model)
    • ElectraConfig configuration class: TFElectraForQuestionAnswering (ELECTRA model)
    • FlaubertConfig configuration class: TFFlaubertForQuestionAnsweringSimple (FlauBERT model)
    • FunnelConfig configuration class: TFFunnelForQuestionAnswering (Funnel Transformer model)
    • GPTJConfig configuration class: TFGPTJForQuestionAnswering (GPT-J model)
    • LayoutLMv3Config configuration class: TFLayoutLMv3ForQuestionAnswering (LayoutLMv3 model)
    • LongformerConfig configuration class: TFLongformerForQuestionAnswering (Longformer model)
    • MPNetConfig configuration class: TFMPNetForQuestionAnswering (MPNet model)
    • MobileBertConfig configuration class: TFMobileBertForQuestionAnswering (MobileBERT model)
    • RemBertConfig configuration class: TFRemBertForQuestionAnswering (RemBERT model)
    • RoFormerConfig configuration class: TFRoFormerForQuestionAnswering (RoFormer model)
    • RobertaConfig configuration class: TFRobertaForQuestionAnswering (RoBERTa model)
    • RobertaPreLayerNormConfig configuration class: TFRobertaPreLayerNormForQuestionAnswering (RoBERTa-PreLayerNorm model)
    • XLMConfig configuration class: TFXLMForQuestionAnsweringSimple (XLM model)
    • XLMRobertaConfig configuration class: TFXLMRobertaForQuestionAnswering (XLM-RoBERTa model)
    • XLNetConfig configuration class: TFXLNetForQuestionAnsweringSimple (XLNet model)
  • attn_implementation (str, optional) — The attention implementation to use in the model (if relevant). Can be any of "eager" (manual implementation of the attention), "sdpa" (using F.scaled_dot_product_attention), or "flash_attention_2" (using Dao-AILab/flash-attention). By default, if available, SDPA will be used for torch>=2.1.1. The default is otherwise the manual "eager" implementation.

Instantiates one of the model classes of the library (with a question answering head) from a configuration.

Note: Loading a model from its configuration file does not load the model weights. It only affects the model’s configuration. Use from_pretrained() to load the model weights.

Examples:

>>> from transformers import AutoConfig, TFAutoModelForQuestionAnswering

>>> # Download configuration from huggingface.co and cache.
>>> config = AutoConfig.from_pretrained("google-bert/bert-base-cased")
>>> model = TFAutoModelForQuestionAnswering.from_config(config)

from_pretrained

< >

( *model_args **kwargs )

Parameters

  • pretrained_model_name_or_path (str or os.PathLike) — Can be either:

    • A string, the model id of a pretrained model hosted inside a model repo on huggingface.co.
    • A path to a directory containing model weights saved using save_pretrained(), e.g., ./my_model_directory/.
    • A path or url to a PyTorch state_dict save file (e.g, ./pt_model/pytorch_model.bin). In this case, from_pt should be set to True and a configuration object should be provided as config argument. This loading path is slower than converting the PyTorch model in a TensorFlow model using the provided conversion scripts and loading the TensorFlow model afterwards.
  • model_args (additional positional arguments, optional) — Will be passed along to the underlying model __init__() method.
  • config (PretrainedConfig, optional) — Configuration for the model to use instead of an automatically loaded configuration. Configuration can be automatically loaded when:

    • The model is a model provided by the library (loaded with the model id string of a pretrained model).
    • The model was saved using save_pretrained() and is reloaded by supplying the save directory.
    • The model is loaded by supplying a local directory as pretrained_model_name_or_path and a configuration JSON file named config.json is found in the directory.
  • cache_dir (str or os.PathLike, optional) — Path to a directory in which a downloaded pretrained model configuration should be cached if the standard cache should not be used.
  • from_pt (bool, optional, defaults to False) — Load the model weights from a PyTorch checkpoint save file (see docstring of pretrained_model_name_or_path argument).
  • force_download (bool, optional, defaults to False) — Whether or not to force the (re-)download of the model weights and configuration files, overriding the cached versions if they exist.
  • resume_download (bool, optional, defaults to False) — Whether or not to delete incompletely received files. Will attempt to resume the download if such a file exists.
  • proxies (Dict[str, str], optional) — A dictionary of proxy servers to use by protocol or endpoint, e.g., {'http': 'foo.bar:3128', 'http://hostname': 'foo.bar:4012'}. The proxies are used on each request.
  • output_loading_info(bool, optional, defaults to False) — Whether ot not to also return a dictionary containing missing keys, unexpected keys and error messages.
  • local_files_only(bool, optional, defaults to False) — Whether or not to only look at local files (e.g., not try downloading the model).
  • revision (str, optional, defaults to "main") — The specific model version to use. It can be a branch name, a tag name, or a commit id, since we use a git-based system for storing models and other artifacts on huggingface.co, so revision can be any identifier allowed by git.
  • trust_remote_code (bool, optional, defaults to False) — Whether or not to allow for custom models defined on the Hub in their own modeling files. This option should only be set to True for repositories you trust and in which you have read the code, as it will execute code present on the Hub on your local machine.
  • code_revision (str, optional, defaults to "main") — The specific revision to use for the code on the Hub, if the code leaves in a different repository than the rest of the model. It can be a branch name, a tag name, or a commit id, since we use a git-based system for storing models and other artifacts on huggingface.co, so revision can be any identifier allowed by git.
  • kwargs (additional keyword arguments, optional) — Can be used to update the configuration object (after it being loaded) and initiate the model (e.g., output_attentions=True). Behaves differently depending on whether a config is provided or automatically loaded:

    • If a configuration is provided with config, **kwargs will be directly passed to the underlying model’s __init__ method (we assume all relevant updates to the configuration have already been done)
    • If a configuration is not provided, kwargs will be first passed to the configuration class initialization function (from_pretrained()). Each key of kwargs that corresponds to a configuration attribute will be used to override said attribute with the supplied kwargs value. Remaining keys that do not correspond to any configuration attribute will be passed to the underlying model’s __init__ function.

Instantiate one of the model classes of the library (with a question answering head) from a pretrained model.

The model class to instantiate is selected based on the model_type property of the config object (either passed as an argument or loaded from pretrained_model_name_or_path if possible), or when it’s missing, by falling back to using pattern matching on pretrained_model_name_or_path:

  • albert — TFAlbertForQuestionAnswering (ALBERT model)
  • bert — TFBertForQuestionAnswering (BERT model)
  • camembert — TFCamembertForQuestionAnswering (CamemBERT model)
  • convbert — TFConvBertForQuestionAnswering (ConvBERT model)
  • deberta — TFDebertaForQuestionAnswering (DeBERTa model)
  • deberta-v2 — TFDebertaV2ForQuestionAnswering (DeBERTa-v2 model)
  • distilbert — TFDistilBertForQuestionAnswering (DistilBERT model)
  • electra — TFElectraForQuestionAnswering (ELECTRA model)
  • flaubert — TFFlaubertForQuestionAnsweringSimple (FlauBERT model)
  • funnel — TFFunnelForQuestionAnswering (Funnel Transformer model)
  • gptj — TFGPTJForQuestionAnswering (GPT-J model)
  • layoutlmv3 — TFLayoutLMv3ForQuestionAnswering (LayoutLMv3 model)
  • longformer — TFLongformerForQuestionAnswering (Longformer model)
  • mobilebert — TFMobileBertForQuestionAnswering (MobileBERT model)
  • mpnet — TFMPNetForQuestionAnswering (MPNet model)
  • rembert — TFRemBertForQuestionAnswering (RemBERT model)
  • roberta — TFRobertaForQuestionAnswering (RoBERTa model)
  • roberta-prelayernorm — TFRobertaPreLayerNormForQuestionAnswering (RoBERTa-PreLayerNorm model)
  • roformer — TFRoFormerForQuestionAnswering (RoFormer model)
  • xlm — TFXLMForQuestionAnsweringSimple (XLM model)
  • xlm-roberta — TFXLMRobertaForQuestionAnswering (XLM-RoBERTa model)
  • xlnet — TFXLNetForQuestionAnsweringSimple (XLNet model)

Examples:

>>> from transformers import AutoConfig, TFAutoModelForQuestionAnswering

>>> # Download model and configuration from huggingface.co and cache.
>>> model = TFAutoModelForQuestionAnswering.from_pretrained("google-bert/bert-base-cased")

>>> # Update configuration during loading
>>> model = TFAutoModelForQuestionAnswering.from_pretrained("google-bert/bert-base-cased", output_attentions=True)
>>> model.config.output_attentions
True

>>> # Loading from a PyTorch checkpoint file instead of a TensorFlow model (slower)
>>> config = AutoConfig.from_pretrained("./pt_model/bert_pt_model_config.json")
>>> model = TFAutoModelForQuestionAnswering.from_pretrained(
...     "./pt_model/bert_pytorch_model.bin", from_pt=True, config=config
... )

FlaxAutoModelForQuestionAnswering

class transformers.FlaxAutoModelForQuestionAnswering

< >

( *args **kwargs )

This is a generic model class that will be instantiated as one of the model classes of the library (with a question answering head) when created with the from_pretrained() class method or the from_config() class method.

This class cannot be instantiated directly using __init__() (throws an error).

from_config

< >

( **kwargs )

Parameters

  • config (PretrainedConfig) — The model class to instantiate is selected based on the configuration class:

    • AlbertConfig configuration class: FlaxAlbertForQuestionAnswering (ALBERT model)
    • BartConfig configuration class: FlaxBartForQuestionAnswering (BART model)
    • BertConfig configuration class: FlaxBertForQuestionAnswering (BERT model)
    • BigBirdConfig configuration class: FlaxBigBirdForQuestionAnswering (BigBird model)
    • DistilBertConfig configuration class: FlaxDistilBertForQuestionAnswering (DistilBERT model)
    • ElectraConfig configuration class: FlaxElectraForQuestionAnswering (ELECTRA model)
    • MBartConfig configuration class: FlaxMBartForQuestionAnswering (mBART model)
    • RoFormerConfig configuration class: FlaxRoFormerForQuestionAnswering (RoFormer model)
    • RobertaConfig configuration class: FlaxRobertaForQuestionAnswering (RoBERTa model)
    • RobertaPreLayerNormConfig configuration class: FlaxRobertaPreLayerNormForQuestionAnswering (RoBERTa-PreLayerNorm model)
    • XLMRobertaConfig configuration class: FlaxXLMRobertaForQuestionAnswering (XLM-RoBERTa model)
  • attn_implementation (str, optional) — The attention implementation to use in the model (if relevant). Can be any of "eager" (manual implementation of the attention), "sdpa" (using F.scaled_dot_product_attention), or "flash_attention_2" (using Dao-AILab/flash-attention). By default, if available, SDPA will be used for torch>=2.1.1. The default is otherwise the manual "eager" implementation.

Instantiates one of the model classes of the library (with a question answering head) from a configuration.

Note: Loading a model from its configuration file does not load the model weights. It only affects the model’s configuration. Use from_pretrained() to load the model weights.

Examples:

>>> from transformers import AutoConfig, FlaxAutoModelForQuestionAnswering

>>> # Download configuration from huggingface.co and cache.
>>> config = AutoConfig.from_pretrained("google-bert/bert-base-cased")
>>> model = FlaxAutoModelForQuestionAnswering.from_config(config)

from_pretrained

< >

( *model_args **kwargs )

Parameters

  • pretrained_model_name_or_path (str or os.PathLike) — Can be either:

    • A string, the model id of a pretrained model hosted inside a model repo on huggingface.co.
    • A path to a directory containing model weights saved using save_pretrained(), e.g., ./my_model_directory/.
    • A path or url to a PyTorch state_dict save file (e.g, ./pt_model/pytorch_model.bin). In this case, from_pt should be set to True and a configuration object should be provided as config argument. This loading path is slower than converting the PyTorch model in a TensorFlow model using the provided conversion scripts and loading the TensorFlow model afterwards.
  • model_args (additional positional arguments, optional) — Will be passed along to the underlying model __init__() method.
  • config (PretrainedConfig, optional) — Configuration for the model to use instead of an automatically loaded configuration. Configuration can be automatically loaded when:

    • The model is a model provided by the library (loaded with the model id string of a pretrained model).
    • The model was saved using save_pretrained() and is reloaded by supplying the save directory.
    • The model is loaded by supplying a local directory as pretrained_model_name_or_path and a configuration JSON file named config.json is found in the directory.
  • cache_dir (str or os.PathLike, optional) — Path to a directory in which a downloaded pretrained model configuration should be cached if the standard cache should not be used.
  • from_pt (bool, optional, defaults to False) — Load the model weights from a PyTorch checkpoint save file (see docstring of pretrained_model_name_or_path argument).
  • force_download (bool, optional, defaults to False) — Whether or not to force the (re-)download of the model weights and configuration files, overriding the cached versions if they exist.
  • resume_download (bool, optional, defaults to False) — Whether or not to delete incompletely received files. Will attempt to resume the download if such a file exists.
  • proxies (Dict[str, str], optional) — A dictionary of proxy servers to use by protocol or endpoint, e.g., {'http': 'foo.bar:3128', 'http://hostname': 'foo.bar:4012'}. The proxies are used on each request.
  • output_loading_info(bool, optional, defaults to False) — Whether ot not to also return a dictionary containing missing keys, unexpected keys and error messages.
  • local_files_only(bool, optional, defaults to False) — Whether or not to only look at local files (e.g., not try downloading the model).
  • revision (str, optional, defaults to "main") — The specific model version to use. It can be a branch name, a tag name, or a commit id, since we use a git-based system for storing models and other artifacts on huggingface.co, so revision can be any identifier allowed by git.
  • trust_remote_code (bool, optional, defaults to False) — Whether or not to allow for custom models defined on the Hub in their own modeling files. This option should only be set to True for repositories you trust and in which you have read the code, as it will execute code present on the Hub on your local machine.
  • code_revision (str, optional, defaults to "main") — The specific revision to use for the code on the Hub, if the code leaves in a different repository than the rest of the model. It can be a branch name, a tag name, or a commit id, since we use a git-based system for storing models and other artifacts on huggingface.co, so revision can be any identifier allowed by git.
  • kwargs (additional keyword arguments, optional) — Can be used to update the configuration object (after it being loaded) and initiate the model (e.g., output_attentions=True). Behaves differently depending on whether a config is provided or automatically loaded:

    • If a configuration is provided with config, **kwargs will be directly passed to the underlying model’s __init__ method (we assume all relevant updates to the configuration have already been done)
    • If a configuration is not provided, kwargs will be first passed to the configuration class initialization function (from_pretrained()). Each key of kwargs that corresponds to a configuration attribute will be used to override said attribute with the supplied kwargs value. Remaining keys that do not correspond to any configuration attribute will be passed to the underlying model’s __init__ function.

Instantiate one of the model classes of the library (with a question answering head) from a pretrained model.

The model class to instantiate is selected based on the model_type property of the config object (either passed as an argument or loaded from pretrained_model_name_or_path if possible), or when it’s missing, by falling back to using pattern matching on pretrained_model_name_or_path:

  • albert — FlaxAlbertForQuestionAnswering (ALBERT model)
  • bart — FlaxBartForQuestionAnswering (BART model)
  • bert — FlaxBertForQuestionAnswering (BERT model)
  • big_bird — FlaxBigBirdForQuestionAnswering (BigBird model)
  • distilbert — FlaxDistilBertForQuestionAnswering (DistilBERT model)
  • electra — FlaxElectraForQuestionAnswering (ELECTRA model)
  • mbart — FlaxMBartForQuestionAnswering (mBART model)
  • roberta — FlaxRobertaForQuestionAnswering (RoBERTa model)
  • roberta-prelayernorm — FlaxRobertaPreLayerNormForQuestionAnswering (RoBERTa-PreLayerNorm model)
  • roformer — FlaxRoFormerForQuestionAnswering (RoFormer model)
  • xlm-roberta — FlaxXLMRobertaForQuestionAnswering (XLM-RoBERTa model)

Examples:

>>> from transformers import AutoConfig, FlaxAutoModelForQuestionAnswering

>>> # Download model and configuration from huggingface.co and cache.
>>> model = FlaxAutoModelForQuestionAnswering.from_pretrained("google-bert/bert-base-cased")

>>> # Update configuration during loading
>>> model = FlaxAutoModelForQuestionAnswering.from_pretrained("google-bert/bert-base-cased", output_attentions=True)
>>> model.config.output_attentions
True

>>> # Loading from a PyTorch checkpoint file instead of a TensorFlow model (slower)
>>> config = AutoConfig.from_pretrained("./pt_model/bert_pt_model_config.json")
>>> model = FlaxAutoModelForQuestionAnswering.from_pretrained(
...     "./pt_model/bert_pytorch_model.bin", from_pt=True, config=config
... )

AutoModelForTextEncoding

class transformers.AutoModelForTextEncoding

< >

( *args **kwargs )

TFAutoModelForTextEncoding

class transformers.TFAutoModelForTextEncoding

< >

( *args **kwargs )

Computer vision

以下の自動クラスは、次のコンピュータービジョンタスクに利用可能です。

AutoModelForDepthEstimation

class transformers.AutoModelForDepthEstimation

< >

( *args **kwargs )

This is a generic model class that will be instantiated as one of the model classes of the library (with a depth estimation head) when created with the from_pretrained() class method or the from_config() class method.

This class cannot be instantiated directly using __init__() (throws an error).

from_config

< >

( **kwargs )

Parameters

  • config (PretrainedConfig) — The model class to instantiate is selected based on the configuration class:

    • DPTConfig configuration class: DPTForDepthEstimation (DPT model)
    • DepthAnythingConfig configuration class: DepthAnythingForDepthEstimation (Depth Anything model)
    • GLPNConfig configuration class: GLPNForDepthEstimation (GLPN model)
  • attn_implementation (str, optional) — The attention implementation to use in the model (if relevant). Can be any of "eager" (manual implementation of the attention), "sdpa" (using F.scaled_dot_product_attention), or "flash_attention_2" (using Dao-AILab/flash-attention). By default, if available, SDPA will be used for torch>=2.1.1. The default is otherwise the manual "eager" implementation.

Instantiates one of the model classes of the library (with a depth estimation head) from a configuration.

Note: Loading a model from its configuration file does not load the model weights. It only affects the model’s configuration. Use from_pretrained() to load the model weights.

Examples:

>>> from transformers import AutoConfig, AutoModelForDepthEstimation

>>> # Download configuration from huggingface.co and cache.
>>> config = AutoConfig.from_pretrained("google-bert/bert-base-cased")
>>> model = AutoModelForDepthEstimation.from_config(config)

from_pretrained

< >

( *model_args **kwargs )

Parameters

  • pretrained_model_name_or_path (str or os.PathLike) — Can be either:

    • A string, the model id of a pretrained model hosted inside a model repo on huggingface.co.
    • A path to a directory containing model weights saved using save_pretrained(), e.g., ./my_model_directory/.
    • A path or url to a tensorflow index checkpoint file (e.g, ./tf_model/model.ckpt.index). In this case, from_tf should be set to True and a configuration object should be provided as config argument. This loading path is slower than converting the TensorFlow checkpoint in a PyTorch model using the provided conversion scripts and loading the PyTorch model afterwards.
  • model_args (additional positional arguments, optional) — Will be passed along to the underlying model __init__() method.
  • config (PretrainedConfig, optional) — Configuration for the model to use instead of an automatically loaded configuration. Configuration can be automatically loaded when:

    • The model is a model provided by the library (loaded with the model id string of a pretrained model).
    • The model was saved using save_pretrained() and is reloaded by supplying the save directory.
    • The model is loaded by supplying a local directory as pretrained_model_name_or_path and a configuration JSON file named config.json is found in the directory.
  • state_dict (Dict[str, torch.Tensor], optional) — A state dictionary to use instead of a state dictionary loaded from saved weights file.

    This option can be used if you want to create a model from a pretrained configuration but load your own weights. In this case though, you should check if using save_pretrained() and from_pretrained() is not a simpler option.

  • cache_dir (str or os.PathLike, optional) — Path to a directory in which a downloaded pretrained model configuration should be cached if the standard cache should not be used.
  • from_tf (bool, optional, defaults to False) — Load the model weights from a TensorFlow checkpoint save file (see docstring of pretrained_model_name_or_path argument).
  • force_download (bool, optional, defaults to False) — Whether or not to force the (re-)download of the model weights and configuration files, overriding the cached versions if they exist.
  • resume_download (bool, optional, defaults to False) — Whether or not to delete incompletely received files. Will attempt to resume the download if such a file exists.
  • proxies (Dict[str, str], optional) — A dictionary of proxy servers to use by protocol or endpoint, e.g., {'http': 'foo.bar:3128', 'http://hostname': 'foo.bar:4012'}. The proxies are used on each request.
  • output_loading_info(bool, optional, defaults to False) — Whether ot not to also return a dictionary containing missing keys, unexpected keys and error messages.
  • local_files_only(bool, optional, defaults to False) — Whether or not to only look at local files (e.g., not try downloading the model).
  • revision (str, optional, defaults to "main") — The specific model version to use. It can be a branch name, a tag name, or a commit id, since we use a git-based system for storing models and other artifacts on huggingface.co, so revision can be any identifier allowed by git.
  • trust_remote_code (bool, optional, defaults to False) — Whether or not to allow for custom models defined on the Hub in their own modeling files. This option should only be set to True for repositories you trust and in which you have read the code, as it will execute code present on the Hub on your local machine.
  • code_revision (str, optional, defaults to "main") — The specific revision to use for the code on the Hub, if the code leaves in a different repository than the rest of the model. It can be a branch name, a tag name, or a commit id, since we use a git-based system for storing models and other artifacts on huggingface.co, so revision can be any identifier allowed by git.
  • kwargs (additional keyword arguments, optional) — Can be used to update the configuration object (after it being loaded) and initiate the model (e.g., output_attentions=True). Behaves differently depending on whether a config is provided or automatically loaded:

    • If a configuration is provided with config, **kwargs will be directly passed to the underlying model’s __init__ method (we assume all relevant updates to the configuration have already been done)
    • If a configuration is not provided, kwargs will be first passed to the configuration class initialization function (from_pretrained()). Each key of kwargs that corresponds to a configuration attribute will be used to override said attribute with the supplied kwargs value. Remaining keys that do not correspond to any configuration attribute will be passed to the underlying model’s __init__ function.

Instantiate one of the model classes of the library (with a depth estimation head) from a pretrained model.

The model class to instantiate is selected based on the model_type property of the config object (either passed as an argument or loaded from pretrained_model_name_or_path if possible), or when it’s missing, by falling back to using pattern matching on pretrained_model_name_or_path:

  • depth_anything — DepthAnythingForDepthEstimation (Depth Anything model)
  • dpt — DPTForDepthEstimation (DPT model)
  • glpn — GLPNForDepthEstimation (GLPN model)

The model is set in evaluation mode by default using model.eval() (so for instance, dropout modules are deactivated). To train the model, you should first set it back in training mode with model.train()

Examples:

>>> from transformers import AutoConfig, AutoModelForDepthEstimation

>>> # Download model and configuration from huggingface.co and cache.
>>> model = AutoModelForDepthEstimation.from_pretrained("google-bert/bert-base-cased")

>>> # Update configuration during loading
>>> model = AutoModelForDepthEstimation.from_pretrained("google-bert/bert-base-cased", output_attentions=True)
>>> model.config.output_attentions
True

>>> # Loading from a TF checkpoint file instead of a PyTorch model (slower)
>>> config = AutoConfig.from_pretrained("./tf_model/bert_tf_model_config.json")
>>> model = AutoModelForDepthEstimation.from_pretrained(
...     "./tf_model/bert_tf_checkpoint.ckpt.index", from_tf=True, config=config
... )

AutoModelForImageClassification

class transformers.AutoModelForImageClassification

< >

( *args **kwargs )

This is a generic model class that will be instantiated as one of the model classes of the library (with a image classification head) when created with the from_pretrained() class method or the from_config() class method.

This class cannot be instantiated directly using __init__() (throws an error).

from_config

< >

( **kwargs )

Parameters

  • config (PretrainedConfig) — The model class to instantiate is selected based on the configuration class:

    • BeitConfig configuration class: BeitForImageClassification (BEiT model)
    • BitConfig configuration class: BitForImageClassification (BiT model)
    • CLIPConfig configuration class: CLIPForImageClassification (CLIP model)
    • ConvNextConfig configuration class: ConvNextForImageClassification (ConvNeXT model)
    • ConvNextV2Config configuration class: ConvNextV2ForImageClassification (ConvNeXTV2 model)
    • CvtConfig configuration class: CvtForImageClassification (CvT model)
    • Data2VecVisionConfig configuration class: Data2VecVisionForImageClassification (Data2VecVision model)
    • DeiTConfig configuration class: DeiTForImageClassification or DeiTForImageClassificationWithTeacher (DeiT model)
    • DinatConfig configuration class: DinatForImageClassification (DiNAT model)
    • Dinov2Config configuration class: Dinov2ForImageClassification (DINOv2 model)
    • EfficientFormerConfig configuration class: EfficientFormerForImageClassification or EfficientFormerForImageClassificationWithTeacher (EfficientFormer model)
    • EfficientNetConfig configuration class: EfficientNetForImageClassification (EfficientNet model)
    • FocalNetConfig configuration class: FocalNetForImageClassification (FocalNet model)
    • ImageGPTConfig configuration class: ImageGPTForImageClassification (ImageGPT model)
    • LevitConfig configuration class: LevitForImageClassification or LevitForImageClassificationWithTeacher (LeViT model)
    • MobileNetV1Config configuration class: MobileNetV1ForImageClassification (MobileNetV1 model)
    • MobileNetV2Config configuration class: MobileNetV2ForImageClassification (MobileNetV2 model)
    • MobileViTConfig configuration class: MobileViTForImageClassification (MobileViT model)
    • MobileViTV2Config configuration class: MobileViTV2ForImageClassification (MobileViTV2 model)
    • NatConfig configuration class: NatForImageClassification (NAT model)
    • PerceiverConfig configuration class: PerceiverForImageClassificationLearned or PerceiverForImageClassificationFourier or PerceiverForImageClassificationConvProcessing (Perceiver model)
    • PoolFormerConfig configuration class: PoolFormerForImageClassification (PoolFormer model)
    • PvtConfig configuration class: PvtForImageClassification (PVT model)
    • PvtV2Config configuration class: PvtV2ForImageClassification (PVTv2 model)
    • RegNetConfig configuration class: RegNetForImageClassification (RegNet model)
    • ResNetConfig configuration class: ResNetForImageClassification (ResNet model)
    • SegformerConfig configuration class: SegformerForImageClassification (SegFormer model)
    • SiglipConfig configuration class: SiglipForImageClassification (SigLIP model)
    • SwiftFormerConfig configuration class: SwiftFormerForImageClassification (SwiftFormer model)
    • SwinConfig configuration class: SwinForImageClassification (Swin Transformer model)
    • Swinv2Config configuration class: Swinv2ForImageClassification (Swin Transformer V2 model)
    • VanConfig configuration class: VanForImageClassification (VAN model)
    • ViTConfig configuration class: ViTForImageClassification (ViT model)
    • ViTHybridConfig configuration class: ViTHybridForImageClassification (ViT Hybrid model)
    • ViTMSNConfig configuration class: ViTMSNForImageClassification (ViTMSN model)
  • attn_implementation (str, optional) — The attention implementation to use in the model (if relevant). Can be any of "eager" (manual implementation of the attention), "sdpa" (using F.scaled_dot_product_attention), or "flash_attention_2" (using Dao-AILab/flash-attention). By default, if available, SDPA will be used for torch>=2.1.1. The default is otherwise the manual "eager" implementation.

Instantiates one of the model classes of the library (with a image classification head) from a configuration.

Note: Loading a model from its configuration file does not load the model weights. It only affects the model’s configuration. Use from_pretrained() to load the model weights.

Examples:

>>> from transformers import AutoConfig, AutoModelForImageClassification

>>> # Download configuration from huggingface.co and cache.
>>> config = AutoConfig.from_pretrained("google-bert/bert-base-cased")
>>> model = AutoModelForImageClassification.from_config(config)

from_pretrained

< >

( *model_args **kwargs )

Parameters

  • pretrained_model_name_or_path (str or os.PathLike) — Can be either:

    • A string, the model id of a pretrained model hosted inside a model repo on huggingface.co.
    • A path to a directory containing model weights saved using save_pretrained(), e.g., ./my_model_directory/.
    • A path or url to a tensorflow index checkpoint file (e.g, ./tf_model/model.ckpt.index). In this case, from_tf should be set to True and a configuration object should be provided as config argument. This loading path is slower than converting the TensorFlow checkpoint in a PyTorch model using the provided conversion scripts and loading the PyTorch model afterwards.
  • model_args (additional positional arguments, optional) — Will be passed along to the underlying model __init__() method.
  • config (PretrainedConfig, optional) — Configuration for the model to use instead of an automatically loaded configuration. Configuration can be automatically loaded when:

    • The model is a model provided by the library (loaded with the model id string of a pretrained model).
    • The model was saved using save_pretrained() and is reloaded by supplying the save directory.
    • The model is loaded by supplying a local directory as pretrained_model_name_or_path and a configuration JSON file named config.json is found in the directory.
  • state_dict (Dict[str, torch.Tensor], optional) — A state dictionary to use instead of a state dictionary loaded from saved weights file.

    This option can be used if you want to create a model from a pretrained configuration but load your own weights. In this case though, you should check if using save_pretrained() and from_pretrained() is not a simpler option.

  • cache_dir (str or os.PathLike, optional) — Path to a directory in which a downloaded pretrained model configuration should be cached if the standard cache should not be used.
  • from_tf (bool, optional, defaults to False) — Load the model weights from a TensorFlow checkpoint save file (see docstring of pretrained_model_name_or_path argument).
  • force_download (bool, optional, defaults to False) — Whether or not to force the (re-)download of the model weights and configuration files, overriding the cached versions if they exist.
  • resume_download (bool, optional, defaults to False) — Whether or not to delete incompletely received files. Will attempt to resume the download if such a file exists.
  • proxies (Dict[str, str], optional) — A dictionary of proxy servers to use by protocol or endpoint, e.g., {'http': 'foo.bar:3128', 'http://hostname': 'foo.bar:4012'}. The proxies are used on each request.
  • output_loading_info(bool, optional, defaults to False) — Whether ot not to also return a dictionary containing missing keys, unexpected keys and error messages.
  • local_files_only(bool, optional, defaults to False) — Whether or not to only look at local files (e.g., not try downloading the model).
  • revision (str, optional, defaults to "main") — The specific model version to use. It can be a branch name, a tag name, or a commit id, since we use a git-based system for storing models and other artifacts on huggingface.co, so revision can be any identifier allowed by git.
  • trust_remote_code (bool, optional, defaults to False) — Whether or not to allow for custom models defined on the Hub in their own modeling files. This option should only be set to True for repositories you trust and in which you have read the code, as it will execute code present on the Hub on your local machine.
  • code_revision (str, optional, defaults to "main") — The specific revision to use for the code on the Hub, if the code leaves in a different repository than the rest of the model. It can be a branch name, a tag name, or a commit id, since we use a git-based system for storing models and other artifacts on huggingface.co, so revision can be any identifier allowed by git.
  • kwargs (additional keyword arguments, optional) — Can be used to update the configuration object (after it being loaded) and initiate the model (e.g., output_attentions=True). Behaves differently depending on whether a config is provided or automatically loaded:

    • If a configuration is provided with config, **kwargs will be directly passed to the underlying model’s __init__ method (we assume all relevant updates to the configuration have already been done)
    • If a configuration is not provided, kwargs will be first passed to the configuration class initialization function (from_pretrained()). Each key of kwargs that corresponds to a configuration attribute will be used to override said attribute with the supplied kwargs value. Remaining keys that do not correspond to any configuration attribute will be passed to the underlying model’s __init__ function.

Instantiate one of the model classes of the library (with a image classification head) from a pretrained model.

The model class to instantiate is selected based on the model_type property of the config object (either passed as an argument or loaded from pretrained_model_name_or_path if possible), or when it’s missing, by falling back to using pattern matching on pretrained_model_name_or_path:

  • beit — BeitForImageClassification (BEiT model)
  • bit — BitForImageClassification (BiT model)
  • clip — CLIPForImageClassification (CLIP model)
  • convnext — ConvNextForImageClassification (ConvNeXT model)
  • convnextv2 — ConvNextV2ForImageClassification (ConvNeXTV2 model)
  • cvt — CvtForImageClassification (CvT model)
  • data2vec-vision — Data2VecVisionForImageClassification (Data2VecVision model)
  • deit — DeiTForImageClassification or DeiTForImageClassificationWithTeacher (DeiT model)
  • dinat — DinatForImageClassification (DiNAT model)
  • dinov2 — Dinov2ForImageClassification (DINOv2 model)
  • efficientformer — EfficientFormerForImageClassification or EfficientFormerForImageClassificationWithTeacher (EfficientFormer model)
  • efficientnet — EfficientNetForImageClassification (EfficientNet model)
  • focalnet — FocalNetForImageClassification (FocalNet model)
  • imagegpt — ImageGPTForImageClassification (ImageGPT model)
  • levit — LevitForImageClassification or LevitForImageClassificationWithTeacher (LeViT model)
  • mobilenet_v1 — MobileNetV1ForImageClassification (MobileNetV1 model)
  • mobilenet_v2 — MobileNetV2ForImageClassification (MobileNetV2 model)
  • mobilevit — MobileViTForImageClassification (MobileViT model)
  • mobilevitv2 — MobileViTV2ForImageClassification (MobileViTV2 model)
  • nat — NatForImageClassification (NAT model)
  • perceiver — PerceiverForImageClassificationLearned or PerceiverForImageClassificationFourier or PerceiverForImageClassificationConvProcessing (Perceiver model)
  • poolformer — PoolFormerForImageClassification (PoolFormer model)
  • pvt — PvtForImageClassification (PVT model)
  • pvt_v2 — PvtV2ForImageClassification (PVTv2 model)
  • regnet — RegNetForImageClassification (RegNet model)
  • resnet — ResNetForImageClassification (ResNet model)
  • segformer — SegformerForImageClassification (SegFormer model)
  • siglip — SiglipForImageClassification (SigLIP model)
  • swiftformer — SwiftFormerForImageClassification (SwiftFormer model)
  • swin — SwinForImageClassification (Swin Transformer model)
  • swinv2 — Swinv2ForImageClassification (Swin Transformer V2 model)
  • van — VanForImageClassification (VAN model)
  • vit — ViTForImageClassification (ViT model)
  • vit_hybrid — ViTHybridForImageClassification (ViT Hybrid model)
  • vit_msn — ViTMSNForImageClassification (ViTMSN model)

The model is set in evaluation mode by default using model.eval() (so for instance, dropout modules are deactivated). To train the model, you should first set it back in training mode with model.train()

Examples:

>>> from transformers import AutoConfig, AutoModelForImageClassification

>>> # Download model and configuration from huggingface.co and cache.
>>> model = AutoModelForImageClassification.from_pretrained("google-bert/bert-base-cased")

>>> # Update configuration during loading
>>> model = AutoModelForImageClassification.from_pretrained("google-bert/bert-base-cased", output_attentions=True)
>>> model.config.output_attentions
True

>>> # Loading from a TF checkpoint file instead of a PyTorch model (slower)
>>> config = AutoConfig.from_pretrained("./tf_model/bert_tf_model_config.json")
>>> model = AutoModelForImageClassification.from_pretrained(
...     "./tf_model/bert_tf_checkpoint.ckpt.index", from_tf=True, config=config
... )

TFAutoModelForImageClassification

class transformers.TFAutoModelForImageClassification

< >

( *args **kwargs )

This is a generic model class that will be instantiated as one of the model classes of the library (with a image classification head) when created with the from_pretrained() class method or the from_config() class method.

This class cannot be instantiated directly using __init__() (throws an error).

from_config

< >

( **kwargs )

Parameters

Instantiates one of the model classes of the library (with a image classification head) from a configuration.

Note: Loading a model from its configuration file does not load the model weights. It only affects the model’s configuration. Use from_pretrained() to load the model weights.

Examples:

>>> from transformers import AutoConfig, TFAutoModelForImageClassification

>>> # Download configuration from huggingface.co and cache.
>>> config = AutoConfig.from_pretrained("google-bert/bert-base-cased")
>>> model = TFAutoModelForImageClassification.from_config(config)

from_pretrained

< >

( *model_args **kwargs )

Parameters

  • pretrained_model_name_or_path (str or os.PathLike) — Can be either:

    • A string, the model id of a pretrained model hosted inside a model repo on huggingface.co.
    • A path to a directory containing model weights saved using save_pretrained(), e.g., ./my_model_directory/.
    • A path or url to a PyTorch state_dict save file (e.g, ./pt_model/pytorch_model.bin). In this case, from_pt should be set to True and a configuration object should be provided as config argument. This loading path is slower than converting the PyTorch model in a TensorFlow model using the provided conversion scripts and loading the TensorFlow model afterwards.
  • model_args (additional positional arguments, optional) — Will be passed along to the underlying model __init__() method.
  • config (PretrainedConfig, optional) — Configuration for the model to use instead of an automatically loaded configuration. Configuration can be automatically loaded when:

    • The model is a model provided by the library (loaded with the model id string of a pretrained model).
    • The model was saved using save_pretrained() and is reloaded by supplying the save directory.
    • The model is loaded by supplying a local directory as pretrained_model_name_or_path and a configuration JSON file named config.json is found in the directory.
  • cache_dir (str or os.PathLike, optional) — Path to a directory in which a downloaded pretrained model configuration should be cached if the standard cache should not be used.
  • from_pt (bool, optional, defaults to False) — Load the model weights from a PyTorch checkpoint save file (see docstring of pretrained_model_name_or_path argument).
  • force_download (bool, optional, defaults to False) — Whether or not to force the (re-)download of the model weights and configuration files, overriding the cached versions if they exist.
  • resume_download (bool, optional, defaults to False) — Whether or not to delete incompletely received files. Will attempt to resume the download if such a file exists.
  • proxies (Dict[str, str], optional) — A dictionary of proxy servers to use by protocol or endpoint, e.g., {'http': 'foo.bar:3128', 'http://hostname': 'foo.bar:4012'}. The proxies are used on each request.
  • output_loading_info(bool, optional, defaults to False) — Whether ot not to also return a dictionary containing missing keys, unexpected keys and error messages.
  • local_files_only(bool, optional, defaults to False) — Whether or not to only look at local files (e.g., not try downloading the model).
  • revision (str, optional, defaults to "main") — The specific model version to use. It can be a branch name, a tag name, or a commit id, since we use a git-based system for storing models and other artifacts on huggingface.co, so revision can be any identifier allowed by git.
  • trust_remote_code (bool, optional, defaults to False) — Whether or not to allow for custom models defined on the Hub in their own modeling files. This option should only be set to True for repositories you trust and in which you have read the code, as it will execute code present on the Hub on your local machine.
  • code_revision (str, optional, defaults to "main") — The specific revision to use for the code on the Hub, if the code leaves in a different repository than the rest of the model. It can be a branch name, a tag name, or a commit id, since we use a git-based system for storing models and other artifacts on huggingface.co, so revision can be any identifier allowed by git.
  • kwargs (additional keyword arguments, optional) — Can be used to update the configuration object (after it being loaded) and initiate the model (e.g., output_attentions=True). Behaves differently depending on whether a config is provided or automatically loaded:

    • If a configuration is provided with config, **kwargs will be directly passed to the underlying model’s __init__ method (we assume all relevant updates to the configuration have already been done)
    • If a configuration is not provided, kwargs will be first passed to the configuration class initialization function (from_pretrained()). Each key of kwargs that corresponds to a configuration attribute will be used to override said attribute with the supplied kwargs value. Remaining keys that do not correspond to any configuration attribute will be passed to the underlying model’s __init__ function.

Instantiate one of the model classes of the library (with a image classification head) from a pretrained model.

The model class to instantiate is selected based on the model_type property of the config object (either passed as an argument or loaded from pretrained_model_name_or_path if possible), or when it’s missing, by falling back to using pattern matching on pretrained_model_name_or_path:

Examples:

>>> from transformers import AutoConfig, TFAutoModelForImageClassification

>>> # Download model and configuration from huggingface.co and cache.
>>> model = TFAutoModelForImageClassification.from_pretrained("google-bert/bert-base-cased")

>>> # Update configuration during loading
>>> model = TFAutoModelForImageClassification.from_pretrained("google-bert/bert-base-cased", output_attentions=True)
>>> model.config.output_attentions
True

>>> # Loading from a PyTorch checkpoint file instead of a TensorFlow model (slower)
>>> config = AutoConfig.from_pretrained("./pt_model/bert_pt_model_config.json")
>>> model = TFAutoModelForImageClassification.from_pretrained(
...     "./pt_model/bert_pytorch_model.bin", from_pt=True, config=config
... )

FlaxAutoModelForImageClassification

class transformers.FlaxAutoModelForImageClassification

< >

( *args **kwargs )

This is a generic model class that will be instantiated as one of the model classes of the library (with a image classification head) when created with the from_pretrained() class method or the from_config() class method.

This class cannot be instantiated directly using __init__() (throws an error).

from_config

< >

( **kwargs )

Parameters

  • config (PretrainedConfig) — The model class to instantiate is selected based on the configuration class:

    • BeitConfig configuration class: FlaxBeitForImageClassification (BEiT model)
    • RegNetConfig configuration class: FlaxRegNetForImageClassification (RegNet model)
    • ResNetConfig configuration class: FlaxResNetForImageClassification (ResNet model)
    • ViTConfig configuration class: FlaxViTForImageClassification (ViT model)
  • attn_implementation (str, optional) — The attention implementation to use in the model (if relevant). Can be any of "eager" (manual implementation of the attention), "sdpa" (using F.scaled_dot_product_attention), or "flash_attention_2" (using Dao-AILab/flash-attention). By default, if available, SDPA will be used for torch>=2.1.1. The default is otherwise the manual "eager" implementation.

Instantiates one of the model classes of the library (with a image classification head) from a configuration.

Note: Loading a model from its configuration file does not load the model weights. It only affects the model’s configuration. Use from_pretrained() to load the model weights.

Examples:

>>> from transformers import AutoConfig, FlaxAutoModelForImageClassification

>>> # Download configuration from huggingface.co and cache.
>>> config = AutoConfig.from_pretrained("google-bert/bert-base-cased")
>>> model = FlaxAutoModelForImageClassification.from_config(config)

from_pretrained

< >

( *model_args **kwargs )

Parameters

  • pretrained_model_name_or_path (str or os.PathLike) — Can be either:

    • A string, the model id of a pretrained model hosted inside a model repo on huggingface.co.
    • A path to a directory containing model weights saved using save_pretrained(), e.g., ./my_model_directory/.
    • A path or url to a PyTorch state_dict save file (e.g, ./pt_model/pytorch_model.bin). In this case, from_pt should be set to True and a configuration object should be provided as config argument. This loading path is slower than converting the PyTorch model in a TensorFlow model using the provided conversion scripts and loading the TensorFlow model afterwards.
  • model_args (additional positional arguments, optional) — Will be passed along to the underlying model __init__() method.
  • config (PretrainedConfig, optional) — Configuration for the model to use instead of an automatically loaded configuration. Configuration can be automatically loaded when:

    • The model is a model provided by the library (loaded with the model id string of a pretrained model).
    • The model was saved using save_pretrained() and is reloaded by supplying the save directory.
    • The model is loaded by supplying a local directory as pretrained_model_name_or_path and a configuration JSON file named config.json is found in the directory.
  • cache_dir (str or os.PathLike, optional) — Path to a directory in which a downloaded pretrained model configuration should be cached if the standard cache should not be used.
  • from_pt (bool, optional, defaults to False) — Load the model weights from a PyTorch checkpoint save file (see docstring of pretrained_model_name_or_path argument).
  • force_download (bool, optional, defaults to False) — Whether or not to force the (re-)download of the model weights and configuration files, overriding the cached versions if they exist.
  • resume_download (bool, optional, defaults to False) — Whether or not to delete incompletely received files. Will attempt to resume the download if such a file exists.
  • proxies (Dict[str, str], optional) — A dictionary of proxy servers to use by protocol or endpoint, e.g., {'http': 'foo.bar:3128', 'http://hostname': 'foo.bar:4012'}. The proxies are used on each request.
  • output_loading_info(bool, optional, defaults to False) — Whether ot not to also return a dictionary containing missing keys, unexpected keys and error messages.
  • local_files_only(bool, optional, defaults to False) — Whether or not to only look at local files (e.g., not try downloading the model).
  • revision (str, optional, defaults to "main") — The specific model version to use. It can be a branch name, a tag name, or a commit id, since we use a git-based system for storing models and other artifacts on huggingface.co, so revision can be any identifier allowed by git.
  • trust_remote_code (bool, optional, defaults to False) — Whether or not to allow for custom models defined on the Hub in their own modeling files. This option should only be set to True for repositories you trust and in which you have read the code, as it will execute code present on the Hub on your local machine.
  • code_revision (str, optional, defaults to "main") — The specific revision to use for the code on the Hub, if the code leaves in a different repository than the rest of the model. It can be a branch name, a tag name, or a commit id, since we use a git-based system for storing models and other artifacts on huggingface.co, so revision can be any identifier allowed by git.
  • kwargs (additional keyword arguments, optional) — Can be used to update the configuration object (after it being loaded) and initiate the model (e.g., output_attentions=True). Behaves differently depending on whether a config is provided or automatically loaded:

    • If a configuration is provided with config, **kwargs will be directly passed to the underlying model’s __init__ method (we assume all relevant updates to the configuration have already been done)
    • If a configuration is not provided, kwargs will be first passed to the configuration class initialization function (from_pretrained()). Each key of kwargs that corresponds to a configuration attribute will be used to override said attribute with the supplied kwargs value. Remaining keys that do not correspond to any configuration attribute will be passed to the underlying model’s __init__ function.

Instantiate one of the model classes of the library (with a image classification head) from a pretrained model.

The model class to instantiate is selected based on the model_type property of the config object (either passed as an argument or loaded from pretrained_model_name_or_path if possible), or when it’s missing, by falling back to using pattern matching on pretrained_model_name_or_path:

  • beit — FlaxBeitForImageClassification (BEiT model)
  • regnet — FlaxRegNetForImageClassification (RegNet model)
  • resnet — FlaxResNetForImageClassification (ResNet model)
  • vit — FlaxViTForImageClassification (ViT model)

Examples:

>>> from transformers import AutoConfig, FlaxAutoModelForImageClassification

>>> # Download model and configuration from huggingface.co and cache.
>>> model = FlaxAutoModelForImageClassification.from_pretrained("google-bert/bert-base-cased")

>>> # Update configuration during loading
>>> model = FlaxAutoModelForImageClassification.from_pretrained("google-bert/bert-base-cased", output_attentions=True)
>>> model.config.output_attentions
True

>>> # Loading from a PyTorch checkpoint file instead of a TensorFlow model (slower)
>>> config = AutoConfig.from_pretrained("./pt_model/bert_pt_model_config.json")
>>> model = FlaxAutoModelForImageClassification.from_pretrained(
...     "./pt_model/bert_pytorch_model.bin", from_pt=True, config=config
... )

AutoModelForVideoClassification

class transformers.AutoModelForVideoClassification

< >

( *args **kwargs )

This is a generic model class that will be instantiated as one of the model classes of the library (with a video classification head) when created with the from_pretrained() class method or the from_config() class method.

This class cannot be instantiated directly using __init__() (throws an error).

from_config

< >

( **kwargs )

Parameters

  • config (PretrainedConfig) — The model class to instantiate is selected based on the configuration class:

    • TimesformerConfig configuration class: TimesformerForVideoClassification (TimeSformer model)
    • VideoMAEConfig configuration class: VideoMAEForVideoClassification (VideoMAE model)
    • VivitConfig configuration class: VivitForVideoClassification (ViViT model)
  • attn_implementation (str, optional) — The attention implementation to use in the model (if relevant). Can be any of "eager" (manual implementation of the attention), "sdpa" (using F.scaled_dot_product_attention), or "flash_attention_2" (using Dao-AILab/flash-attention). By default, if available, SDPA will be used for torch>=2.1.1. The default is otherwise the manual "eager" implementation.

Instantiates one of the model classes of the library (with a video classification head) from a configuration.

Note: Loading a model from its configuration file does not load the model weights. It only affects the model’s configuration. Use from_pretrained() to load the model weights.

Examples:

>>> from transformers import AutoConfig, AutoModelForVideoClassification

>>> # Download configuration from huggingface.co and cache.
>>> config = AutoConfig.from_pretrained("google-bert/bert-base-cased")
>>> model = AutoModelForVideoClassification.from_config(config)

from_pretrained

< >

( *model_args **kwargs )

Parameters

  • pretrained_model_name_or_path (str or os.PathLike) — Can be either:

    • A string, the model id of a pretrained model hosted inside a model repo on huggingface.co.
    • A path to a directory containing model weights saved using save_pretrained(), e.g., ./my_model_directory/.
    • A path or url to a tensorflow index checkpoint file (e.g, ./tf_model/model.ckpt.index). In this case, from_tf should be set to True and a configuration object should be provided as config argument. This loading path is slower than converting the TensorFlow checkpoint in a PyTorch model using the provided conversion scripts and loading the PyTorch model afterwards.
  • model_args (additional positional arguments, optional) — Will be passed along to the underlying model __init__() method.
  • config (PretrainedConfig, optional) — Configuration for the model to use instead of an automatically loaded configuration. Configuration can be automatically loaded when:

    • The model is a model provided by the library (loaded with the model id string of a pretrained model).
    • The model was saved using save_pretrained() and is reloaded by supplying the save directory.
    • The model is loaded by supplying a local directory as pretrained_model_name_or_path and a configuration JSON file named config.json is found in the directory.
  • state_dict (Dict[str, torch.Tensor], optional) — A state dictionary to use instead of a state dictionary loaded from saved weights file.

    This option can be used if you want to create a model from a pretrained configuration but load your own weights. In this case though, you should check if using save_pretrained() and from_pretrained() is not a simpler option.

  • cache_dir (str or os.PathLike, optional) — Path to a directory in which a downloaded pretrained model configuration should be cached if the standard cache should not be used.
  • from_tf (bool, optional, defaults to False) — Load the model weights from a TensorFlow checkpoint save file (see docstring of pretrained_model_name_or_path argument).
  • force_download (bool, optional, defaults to False) — Whether or not to force the (re-)download of the model weights and configuration files, overriding the cached versions if they exist.
  • resume_download (bool, optional, defaults to False) — Whether or not to delete incompletely received files. Will attempt to resume the download if such a file exists.
  • proxies (Dict[str, str], optional) — A dictionary of proxy servers to use by protocol or endpoint, e.g., {'http': 'foo.bar:3128', 'http://hostname': 'foo.bar:4012'}. The proxies are used on each request.
  • output_loading_info(bool, optional, defaults to False) — Whether ot not to also return a dictionary containing missing keys, unexpected keys and error messages.
  • local_files_only(bool, optional, defaults to False) — Whether or not to only look at local files (e.g., not try downloading the model).
  • revision (str, optional, defaults to "main") — The specific model version to use. It can be a branch name, a tag name, or a commit id, since we use a git-based system for storing models and other artifacts on huggingface.co, so revision can be any identifier allowed by git.
  • trust_remote_code (bool, optional, defaults to False) — Whether or not to allow for custom models defined on the Hub in their own modeling files. This option should only be set to True for repositories you trust and in which you have read the code, as it will execute code present on the Hub on your local machine.
  • code_revision (str, optional, defaults to "main") — The specific revision to use for the code on the Hub, if the code leaves in a different repository than the rest of the model. It can be a branch name, a tag name, or a commit id, since we use a git-based system for storing models and other artifacts on huggingface.co, so revision can be any identifier allowed by git.
  • kwargs (additional keyword arguments, optional) — Can be used to update the configuration object (after it being loaded) and initiate the model (e.g., output_attentions=True). Behaves differently depending on whether a config is provided or automatically loaded:

    • If a configuration is provided with config, **kwargs will be directly passed to the underlying model’s __init__ method (we assume all relevant updates to the configuration have already been done)
    • If a configuration is not provided, kwargs will be first passed to the configuration class initialization function (from_pretrained()). Each key of kwargs that corresponds to a configuration attribute will be used to override said attribute with the supplied kwargs value. Remaining keys that do not correspond to any configuration attribute will be passed to the underlying model’s __init__ function.

Instantiate one of the model classes of the library (with a video classification head) from a pretrained model.

The model class to instantiate is selected based on the model_type property of the config object (either passed as an argument or loaded from pretrained_model_name_or_path if possible), or when it’s missing, by falling back to using pattern matching on pretrained_model_name_or_path:

  • timesformer — TimesformerForVideoClassification (TimeSformer model)
  • videomae — VideoMAEForVideoClassification (VideoMAE model)
  • vivit — VivitForVideoClassification (ViViT model)

The model is set in evaluation mode by default using model.eval() (so for instance, dropout modules are deactivated). To train the model, you should first set it back in training mode with model.train()

Examples:

>>> from transformers import AutoConfig, AutoModelForVideoClassification

>>> # Download model and configuration from huggingface.co and cache.
>>> model = AutoModelForVideoClassification.from_pretrained("google-bert/bert-base-cased")

>>> # Update configuration during loading
>>> model = AutoModelForVideoClassification.from_pretrained("google-bert/bert-base-cased", output_attentions=True)
>>> model.config.output_attentions
True

>>> # Loading from a TF checkpoint file instead of a PyTorch model (slower)
>>> config = AutoConfig.from_pretrained("./tf_model/bert_tf_model_config.json")
>>> model = AutoModelForVideoClassification.from_pretrained(
...     "./tf_model/bert_tf_checkpoint.ckpt.index", from_tf=True, config=config
... )

AutoModelForMaskedImageModeling

class transformers.AutoModelForMaskedImageModeling

< >

( *args **kwargs )

This is a generic model class that will be instantiated as one of the model classes of the library (with a masked image modeling head) when created with the from_pretrained() class method or the from_config() class method.

This class cannot be instantiated directly using __init__() (throws an error).

from_config

< >

( **kwargs )

Parameters

  • config (PretrainedConfig) — The model class to instantiate is selected based on the configuration class:

    • DeiTConfig configuration class: DeiTForMaskedImageModeling (DeiT model)
    • FocalNetConfig configuration class: FocalNetForMaskedImageModeling (FocalNet model)
    • SwinConfig configuration class: SwinForMaskedImageModeling (Swin Transformer model)
    • Swinv2Config configuration class: Swinv2ForMaskedImageModeling (Swin Transformer V2 model)
    • ViTConfig configuration class: ViTForMaskedImageModeling (ViT model)
  • attn_implementation (str, optional) — The attention implementation to use in the model (if relevant). Can be any of "eager" (manual implementation of the attention), "sdpa" (using F.scaled_dot_product_attention), or "flash_attention_2" (using Dao-AILab/flash-attention). By default, if available, SDPA will be used for torch>=2.1.1. The default is otherwise the manual "eager" implementation.

Instantiates one of the model classes of the library (with a masked image modeling head) from a configuration.

Note: Loading a model from its configuration file does not load the model weights. It only affects the model’s configuration. Use from_pretrained() to load the model weights.

Examples:

>>> from transformers import AutoConfig, AutoModelForMaskedImageModeling

>>> # Download configuration from huggingface.co and cache.
>>> config = AutoConfig.from_pretrained("google-bert/bert-base-cased")
>>> model = AutoModelForMaskedImageModeling.from_config(config)

from_pretrained

< >

( *model_args **kwargs )

Parameters

  • pretrained_model_name_or_path (str or os.PathLike) — Can be either:

    • A string, the model id of a pretrained model hosted inside a model repo on huggingface.co.
    • A path to a directory containing model weights saved using save_pretrained(), e.g., ./my_model_directory/.
    • A path or url to a tensorflow index checkpoint file (e.g, ./tf_model/model.ckpt.index). In this case, from_tf should be set to True and a configuration object should be provided as config argument. This loading path is slower than converting the TensorFlow checkpoint in a PyTorch model using the provided conversion scripts and loading the PyTorch model afterwards.
  • model_args (additional positional arguments, optional) — Will be passed along to the underlying model __init__() method.
  • config (PretrainedConfig, optional) — Configuration for the model to use instead of an automatically loaded configuration. Configuration can be automatically loaded when:

    • The model is a model provided by the library (loaded with the model id string of a pretrained model).
    • The model was saved using save_pretrained() and is reloaded by supplying the save directory.
    • The model is loaded by supplying a local directory as pretrained_model_name_or_path and a configuration JSON file named config.json is found in the directory.
  • state_dict (Dict[str, torch.Tensor], optional) — A state dictionary to use instead of a state dictionary loaded from saved weights file.

    This option can be used if you want to create a model from a pretrained configuration but load your own weights. In this case though, you should check if using save_pretrained() and from_pretrained() is not a simpler option.

  • cache_dir (str or os.PathLike, optional) — Path to a directory in which a downloaded pretrained model configuration should be cached if the standard cache should not be used.
  • from_tf (bool, optional, defaults to False) — Load the model weights from a TensorFlow checkpoint save file (see docstring of pretrained_model_name_or_path argument).
  • force_download (bool, optional, defaults to False) — Whether or not to force the (re-)download of the model weights and configuration files, overriding the cached versions if they exist.
  • resume_download (bool, optional, defaults to False) — Whether or not to delete incompletely received files. Will attempt to resume the download if such a file exists.
  • proxies (Dict[str, str], optional) — A dictionary of proxy servers to use by protocol or endpoint, e.g., {'http': 'foo.bar:3128', 'http://hostname': 'foo.bar:4012'}. The proxies are used on each request.
  • output_loading_info(bool, optional, defaults to False) — Whether ot not to also return a dictionary containing missing keys, unexpected keys and error messages.
  • local_files_only(bool, optional, defaults to False) — Whether or not to only look at local files (e.g., not try downloading the model).
  • revision (str, optional, defaults to "main") — The specific model version to use. It can be a branch name, a tag name, or a commit id, since we use a git-based system for storing models and other artifacts on huggingface.co, so revision can be any identifier allowed by git.
  • trust_remote_code (bool, optional, defaults to False) — Whether or not to allow for custom models defined on the Hub in their own modeling files. This option should only be set to True for repositories you trust and in which you have read the code, as it will execute code present on the Hub on your local machine.
  • code_revision (str, optional, defaults to "main") — The specific revision to use for the code on the Hub, if the code leaves in a different repository than the rest of the model. It can be a branch name, a tag name, or a commit id, since we use a git-based system for storing models and other artifacts on huggingface.co, so revision can be any identifier allowed by git.
  • kwargs (additional keyword arguments, optional) — Can be used to update the configuration object (after it being loaded) and initiate the model (e.g., output_attentions=True). Behaves differently depending on whether a config is provided or automatically loaded:

    • If a configuration is provided with config, **kwargs will be directly passed to the underlying model’s __init__ method (we assume all relevant updates to the configuration have already been done)
    • If a configuration is not provided, kwargs will be first passed to the configuration class initialization function (from_pretrained()). Each key of kwargs that corresponds to a configuration attribute will be used to override said attribute with the supplied kwargs value. Remaining keys that do not correspond to any configuration attribute will be passed to the underlying model’s __init__ function.

Instantiate one of the model classes of the library (with a masked image modeling head) from a pretrained model.

The model class to instantiate is selected based on the model_type property of the config object (either passed as an argument or loaded from pretrained_model_name_or_path if possible), or when it’s missing, by falling back to using pattern matching on pretrained_model_name_or_path:

  • deit — DeiTForMaskedImageModeling (DeiT model)
  • focalnet — FocalNetForMaskedImageModeling (FocalNet model)
  • swin — SwinForMaskedImageModeling (Swin Transformer model)
  • swinv2 — Swinv2ForMaskedImageModeling (Swin Transformer V2 model)
  • vit — ViTForMaskedImageModeling (ViT model)

The model is set in evaluation mode by default using model.eval() (so for instance, dropout modules are deactivated). To train the model, you should first set it back in training mode with model.train()

Examples:

>>> from transformers import AutoConfig, AutoModelForMaskedImageModeling

>>> # Download model and configuration from huggingface.co and cache.
>>> model = AutoModelForMaskedImageModeling.from_pretrained("google-bert/bert-base-cased")

>>> # Update configuration during loading
>>> model = AutoModelForMaskedImageModeling.from_pretrained("google-bert/bert-base-cased", output_attentions=True)
>>> model.config.output_attentions
True

>>> # Loading from a TF checkpoint file instead of a PyTorch model (slower)
>>> config = AutoConfig.from_pretrained("./tf_model/bert_tf_model_config.json")
>>> model = AutoModelForMaskedImageModeling.from_pretrained(
...     "./tf_model/bert_tf_checkpoint.ckpt.index", from_tf=True, config=config
... )

TFAutoModelForMaskedImageModeling

class transformers.TFAutoModelForMaskedImageModeling

< >

( *args **kwargs )

This is a generic model class that will be instantiated as one of the model classes of the library (with a masked image modeling head) when created with the from_pretrained() class method or the from_config() class method.

This class cannot be instantiated directly using __init__() (throws an error).

from_config

< >

( **kwargs )

Parameters

  • config (PretrainedConfig) — The model class to instantiate is selected based on the configuration class:
  • attn_implementation (str, optional) — The attention implementation to use in the model (if relevant). Can be any of "eager" (manual implementation of the attention), "sdpa" (using F.scaled_dot_product_attention), or "flash_attention_2" (using Dao-AILab/flash-attention). By default, if available, SDPA will be used for torch>=2.1.1. The default is otherwise the manual "eager" implementation.

Instantiates one of the model classes of the library (with a masked image modeling head) from a configuration.

Note: Loading a model from its configuration file does not load the model weights. It only affects the model’s configuration. Use from_pretrained() to load the model weights.

Examples:

>>> from transformers import AutoConfig, TFAutoModelForMaskedImageModeling

>>> # Download configuration from huggingface.co and cache.
>>> config = AutoConfig.from_pretrained("google-bert/bert-base-cased")
>>> model = TFAutoModelForMaskedImageModeling.from_config(config)

from_pretrained

< >

( *model_args **kwargs )

Parameters

  • pretrained_model_name_or_path (str or os.PathLike) — Can be either:

    • A string, the model id of a pretrained model hosted inside a model repo on huggingface.co.
    • A path to a directory containing model weights saved using save_pretrained(), e.g., ./my_model_directory/.
    • A path or url to a PyTorch state_dict save file (e.g, ./pt_model/pytorch_model.bin). In this case, from_pt should be set to True and a configuration object should be provided as config argument. This loading path is slower than converting the PyTorch model in a TensorFlow model using the provided conversion scripts and loading the TensorFlow model afterwards.
  • model_args (additional positional arguments, optional) — Will be passed along to the underlying model __init__() method.
  • config (PretrainedConfig, optional) — Configuration for the model to use instead of an automatically loaded configuration. Configuration can be automatically loaded when:

    • The model is a model provided by the library (loaded with the model id string of a pretrained model).
    • The model was saved using save_pretrained() and is reloaded by supplying the save directory.
    • The model is loaded by supplying a local directory as pretrained_model_name_or_path and a configuration JSON file named config.json is found in the directory.
  • cache_dir (str or os.PathLike, optional) — Path to a directory in which a downloaded pretrained model configuration should be cached if the standard cache should not be used.
  • from_pt (bool, optional, defaults to False) — Load the model weights from a PyTorch checkpoint save file (see docstring of pretrained_model_name_or_path argument).
  • force_download (bool, optional, defaults to False) — Whether or not to force the (re-)download of the model weights and configuration files, overriding the cached versions if they exist.
  • resume_download (bool, optional, defaults to False) — Whether or not to delete incompletely received files. Will attempt to resume the download if such a file exists.
  • proxies (Dict[str, str], optional) — A dictionary of proxy servers to use by protocol or endpoint, e.g., {'http': 'foo.bar:3128', 'http://hostname': 'foo.bar:4012'}. The proxies are used on each request.
  • output_loading_info(bool, optional, defaults to False) — Whether ot not to also return a dictionary containing missing keys, unexpected keys and error messages.
  • local_files_only(bool, optional, defaults to False) — Whether or not to only look at local files (e.g., not try downloading the model).
  • revision (str, optional, defaults to "main") — The specific model version to use. It can be a branch name, a tag name, or a commit id, since we use a git-based system for storing models and other artifacts on huggingface.co, so revision can be any identifier allowed by git.
  • trust_remote_code (bool, optional, defaults to False) — Whether or not to allow for custom models defined on the Hub in their own modeling files. This option should only be set to True for repositories you trust and in which you have read the code, as it will execute code present on the Hub on your local machine.
  • code_revision (str, optional, defaults to "main") — The specific revision to use for the code on the Hub, if the code leaves in a different repository than the rest of the model. It can be a branch name, a tag name, or a commit id, since we use a git-based system for storing models and other artifacts on huggingface.co, so revision can be any identifier allowed by git.
  • kwargs (additional keyword arguments, optional) — Can be used to update the configuration object (after it being loaded) and initiate the model (e.g., output_attentions=True). Behaves differently depending on whether a config is provided or automatically loaded:

    • If a configuration is provided with config, **kwargs will be directly passed to the underlying model’s __init__ method (we assume all relevant updates to the configuration have already been done)
    • If a configuration is not provided, kwargs will be first passed to the configuration class initialization function (from_pretrained()). Each key of kwargs that corresponds to a configuration attribute will be used to override said attribute with the supplied kwargs value. Remaining keys that do not correspond to any configuration attribute will be passed to the underlying model’s __init__ function.

Instantiate one of the model classes of the library (with a masked image modeling head) from a pretrained model.

The model class to instantiate is selected based on the model_type property of the config object (either passed as an argument or loaded from pretrained_model_name_or_path if possible), or when it’s missing, by falling back to using pattern matching on pretrained_model_name_or_path:

Examples:

>>> from transformers import AutoConfig, TFAutoModelForMaskedImageModeling

>>> # Download model and configuration from huggingface.co and cache.
>>> model = TFAutoModelForMaskedImageModeling.from_pretrained("google-bert/bert-base-cased")

>>> # Update configuration during loading
>>> model = TFAutoModelForMaskedImageModeling.from_pretrained("google-bert/bert-base-cased", output_attentions=True)
>>> model.config.output_attentions
True

>>> # Loading from a PyTorch checkpoint file instead of a TensorFlow model (slower)
>>> config = AutoConfig.from_pretrained("./pt_model/bert_pt_model_config.json")
>>> model = TFAutoModelForMaskedImageModeling.from_pretrained(
...     "./pt_model/bert_pytorch_model.bin", from_pt=True, config=config
... )

AutoModelForObjectDetection

class transformers.AutoModelForObjectDetection

< >

( *args **kwargs )

This is a generic model class that will be instantiated as one of the model classes of the library (with a object detection head) when created with the from_pretrained() class method or the from_config() class method.

This class cannot be instantiated directly using __init__() (throws an error).

from_config

< >

( **kwargs )

Parameters

Instantiates one of the model classes of the library (with a object detection head) from a configuration.

Note: Loading a model from its configuration file does not load the model weights. It only affects the model’s configuration. Use from_pretrained() to load the model weights.

Examples:

>>> from transformers import AutoConfig, AutoModelForObjectDetection

>>> # Download configuration from huggingface.co and cache.
>>> config = AutoConfig.from_pretrained("google-bert/bert-base-cased")
>>> model = AutoModelForObjectDetection.from_config(config)

from_pretrained

< >

( *model_args **kwargs )

Parameters

  • pretrained_model_name_or_path (str or os.PathLike) — Can be either:

    • A string, the model id of a pretrained model hosted inside a model repo on huggingface.co.
    • A path to a directory containing model weights saved using save_pretrained(), e.g., ./my_model_directory/.
    • A path or url to a tensorflow index checkpoint file (e.g, ./tf_model/model.ckpt.index). In this case, from_tf should be set to True and a configuration object should be provided as config argument. This loading path is slower than converting the TensorFlow checkpoint in a PyTorch model using the provided conversion scripts and loading the PyTorch model afterwards.
  • model_args (additional positional arguments, optional) — Will be passed along to the underlying model __init__() method.
  • config (PretrainedConfig, optional) — Configuration for the model to use instead of an automatically loaded configuration. Configuration can be automatically loaded when:

    • The model is a model provided by the library (loaded with the model id string of a pretrained model).
    • The model was saved using save_pretrained() and is reloaded by supplying the save directory.
    • The model is loaded by supplying a local directory as pretrained_model_name_or_path and a configuration JSON file named config.json is found in the directory.
  • state_dict (Dict[str, torch.Tensor], optional) — A state dictionary to use instead of a state dictionary loaded from saved weights file.

    This option can be used if you want to create a model from a pretrained configuration but load your own weights. In this case though, you should check if using save_pretrained() and from_pretrained() is not a simpler option.

  • cache_dir (str or os.PathLike, optional) — Path to a directory in which a downloaded pretrained model configuration should be cached if the standard cache should not be used.
  • from_tf (bool, optional, defaults to False) — Load the model weights from a TensorFlow checkpoint save file (see docstring of pretrained_model_name_or_path argument).
  • force_download (bool, optional, defaults to False) — Whether or not to force the (re-)download of the model weights and configuration files, overriding the cached versions if they exist.
  • resume_download (bool, optional, defaults to False) — Whether or not to delete incompletely received files. Will attempt to resume the download if such a file exists.
  • proxies (Dict[str, str], optional) — A dictionary of proxy servers to use by protocol or endpoint, e.g., {'http': 'foo.bar:3128', 'http://hostname': 'foo.bar:4012'}. The proxies are used on each request.
  • output_loading_info(bool, optional, defaults to False) — Whether ot not to also return a dictionary containing missing keys, unexpected keys and error messages.
  • local_files_only(bool, optional, defaults to False) — Whether or not to only look at local files (e.g., not try downloading the model).
  • revision (str, optional, defaults to "main") — The specific model version to use. It can be a branch name, a tag name, or a commit id, since we use a git-based system for storing models and other artifacts on huggingface.co, so revision can be any identifier allowed by git.
  • trust_remote_code (bool, optional, defaults to False) — Whether or not to allow for custom models defined on the Hub in their own modeling files. This option should only be set to True for repositories you trust and in which you have read the code, as it will execute code present on the Hub on your local machine.
  • code_revision (str, optional, defaults to "main") — The specific revision to use for the code on the Hub, if the code leaves in a different repository than the rest of the model. It can be a branch name, a tag name, or a commit id, since we use a git-based system for storing models and other artifacts on huggingface.co, so revision can be any identifier allowed by git.
  • kwargs (additional keyword arguments, optional) — Can be used to update the configuration object (after it being loaded) and initiate the model (e.g., output_attentions=True). Behaves differently depending on whether a config is provided or automatically loaded:

    • If a configuration is provided with config, **kwargs will be directly passed to the underlying model’s __init__ method (we assume all relevant updates to the configuration have already been done)
    • If a configuration is not provided, kwargs will be first passed to the configuration class initialization function (from_pretrained()). Each key of kwargs that corresponds to a configuration attribute will be used to override said attribute with the supplied kwargs value. Remaining keys that do not correspond to any configuration attribute will be passed to the underlying model’s __init__ function.

Instantiate one of the model classes of the library (with a object detection head) from a pretrained model.

The model class to instantiate is selected based on the model_type property of the config object (either passed as an argument or loaded from pretrained_model_name_or_path if possible), or when it’s missing, by falling back to using pattern matching on pretrained_model_name_or_path:

The model is set in evaluation mode by default using model.eval() (so for instance, dropout modules are deactivated). To train the model, you should first set it back in training mode with model.train()

Examples:

>>> from transformers import AutoConfig, AutoModelForObjectDetection

>>> # Download model and configuration from huggingface.co and cache.
>>> model = AutoModelForObjectDetection.from_pretrained("google-bert/bert-base-cased")

>>> # Update configuration during loading
>>> model = AutoModelForObjectDetection.from_pretrained("google-bert/bert-base-cased", output_attentions=True)
>>> model.config.output_attentions
True

>>> # Loading from a TF checkpoint file instead of a PyTorch model (slower)
>>> config = AutoConfig.from_pretrained("./tf_model/bert_tf_model_config.json")
>>> model = AutoModelForObjectDetection.from_pretrained(
...     "./tf_model/bert_tf_checkpoint.ckpt.index", from_tf=True, config=config
... )

AutoModelForImageSegmentation

class transformers.AutoModelForImageSegmentation

< >

( *args **kwargs )

This is a generic model class that will be instantiated as one of the model classes of the library (with a image segmentation head) when created with the from_pretrained() class method or the from_config() class method.

This class cannot be instantiated directly using __init__() (throws an error).

from_config

< >

( **kwargs )

Parameters

  • config (PretrainedConfig) — The model class to instantiate is selected based on the configuration class:
  • attn_implementation (str, optional) — The attention implementation to use in the model (if relevant). Can be any of "eager" (manual implementation of the attention), "sdpa" (using F.scaled_dot_product_attention), or "flash_attention_2" (using Dao-AILab/flash-attention). By default, if available, SDPA will be used for torch>=2.1.1. The default is otherwise the manual "eager" implementation.

Instantiates one of the model classes of the library (with a image segmentation head) from a configuration.

Note: Loading a model from its configuration file does not load the model weights. It only affects the model’s configuration. Use from_pretrained() to load the model weights.

Examples:

>>> from transformers import AutoConfig, AutoModelForImageSegmentation

>>> # Download configuration from huggingface.co and cache.
>>> config = AutoConfig.from_pretrained("google-bert/bert-base-cased")
>>> model = AutoModelForImageSegmentation.from_config(config)

from_pretrained

< >

( *model_args **kwargs )

Parameters

  • pretrained_model_name_or_path (str or os.PathLike) — Can be either:

    • A string, the model id of a pretrained model hosted inside a model repo on huggingface.co.
    • A path to a directory containing model weights saved using save_pretrained(), e.g., ./my_model_directory/.
    • A path or url to a tensorflow index checkpoint file (e.g, ./tf_model/model.ckpt.index). In this case, from_tf should be set to True and a configuration object should be provided as config argument. This loading path is slower than converting the TensorFlow checkpoint in a PyTorch model using the provided conversion scripts and loading the PyTorch model afterwards.
  • model_args (additional positional arguments, optional) — Will be passed along to the underlying model __init__() method.
  • config (PretrainedConfig, optional) — Configuration for the model to use instead of an automatically loaded configuration. Configuration can be automatically loaded when:

    • The model is a model provided by the library (loaded with the model id string of a pretrained model).
    • The model was saved using save_pretrained() and is reloaded by supplying the save directory.
    • The model is loaded by supplying a local directory as pretrained_model_name_or_path and a configuration JSON file named config.json is found in the directory.
  • state_dict (Dict[str, torch.Tensor], optional) — A state dictionary to use instead of a state dictionary loaded from saved weights file.

    This option can be used if you want to create a model from a pretrained configuration but load your own weights. In this case though, you should check if using save_pretrained() and from_pretrained() is not a simpler option.

  • cache_dir (str or os.PathLike, optional) — Path to a directory in which a downloaded pretrained model configuration should be cached if the standard cache should not be used.
  • from_tf (bool, optional, defaults to False) — Load the model weights from a TensorFlow checkpoint save file (see docstring of pretrained_model_name_or_path argument).
  • force_download (bool, optional, defaults to False) — Whether or not to force the (re-)download of the model weights and configuration files, overriding the cached versions if they exist.
  • resume_download (bool, optional, defaults to False) — Whether or not to delete incompletely received files. Will attempt to resume the download if such a file exists.
  • proxies (Dict[str, str], optional) — A dictionary of proxy servers to use by protocol or endpoint, e.g., {'http': 'foo.bar:3128', 'http://hostname': 'foo.bar:4012'}. The proxies are used on each request.
  • output_loading_info(bool, optional, defaults to False) — Whether ot not to also return a dictionary containing missing keys, unexpected keys and error messages.
  • local_files_only(bool, optional, defaults to False) — Whether or not to only look at local files (e.g., not try downloading the model).
  • revision (str, optional, defaults to "main") — The specific model version to use. It can be a branch name, a tag name, or a commit id, since we use a git-based system for storing models and other artifacts on huggingface.co, so revision can be any identifier allowed by git.
  • trust_remote_code (bool, optional, defaults to False) — Whether or not to allow for custom models defined on the Hub in their own modeling files. This option should only be set to True for repositories you trust and in which you have read the code, as it will execute code present on the Hub on your local machine.
  • code_revision (str, optional, defaults to "main") — The specific revision to use for the code on the Hub, if the code leaves in a different repository than the rest of the model. It can be a branch name, a tag name, or a commit id, since we use a git-based system for storing models and other artifacts on huggingface.co, so revision can be any identifier allowed by git.
  • kwargs (additional keyword arguments, optional) — Can be used to update the configuration object (after it being loaded) and initiate the model (e.g., output_attentions=True). Behaves differently depending on whether a config is provided or automatically loaded:

    • If a configuration is provided with config, **kwargs will be directly passed to the underlying model’s __init__ method (we assume all relevant updates to the configuration have already been done)
    • If a configuration is not provided, kwargs will be first passed to the configuration class initialization function (from_pretrained()). Each key of kwargs that corresponds to a configuration attribute will be used to override said attribute with the supplied kwargs value. Remaining keys that do not correspond to any configuration attribute will be passed to the underlying model’s __init__ function.

Instantiate one of the model classes of the library (with a image segmentation head) from a pretrained model.

The model class to instantiate is selected based on the model_type property of the config object (either passed as an argument or loaded from pretrained_model_name_or_path if possible), or when it’s missing, by falling back to using pattern matching on pretrained_model_name_or_path:

The model is set in evaluation mode by default using model.eval() (so for instance, dropout modules are deactivated). To train the model, you should first set it back in training mode with model.train()

Examples:

>>> from transformers import AutoConfig, AutoModelForImageSegmentation

>>> # Download model and configuration from huggingface.co and cache.
>>> model = AutoModelForImageSegmentation.from_pretrained("google-bert/bert-base-cased")

>>> # Update configuration during loading
>>> model = AutoModelForImageSegmentation.from_pretrained("google-bert/bert-base-cased", output_attentions=True)
>>> model.config.output_attentions
True

>>> # Loading from a TF checkpoint file instead of a PyTorch model (slower)
>>> config = AutoConfig.from_pretrained("./tf_model/bert_tf_model_config.json")
>>> model = AutoModelForImageSegmentation.from_pretrained(
...     "./tf_model/bert_tf_checkpoint.ckpt.index", from_tf=True, config=config
... )

AutoModelForImageToImage

class transformers.AutoModelForImageToImage

< >

( *args **kwargs )

AutoModelForSemanticSegmentation

class transformers.AutoModelForSemanticSegmentation

< >

( *args **kwargs )

This is a generic model class that will be instantiated as one of the model classes of the library (with a semantic segmentation head) when created with the from_pretrained() class method or the from_config() class method.

This class cannot be instantiated directly using __init__() (throws an error).

from_config

< >

( **kwargs )

Parameters

  • config (PretrainedConfig) — The model class to instantiate is selected based on the configuration class:

    • BeitConfig configuration class: BeitForSemanticSegmentation (BEiT model)
    • DPTConfig configuration class: DPTForSemanticSegmentation (DPT model)
    • Data2VecVisionConfig configuration class: Data2VecVisionForSemanticSegmentation (Data2VecVision model)
    • MobileNetV2Config configuration class: MobileNetV2ForSemanticSegmentation (MobileNetV2 model)
    • MobileViTConfig configuration class: MobileViTForSemanticSegmentation (MobileViT model)
    • MobileViTV2Config configuration class: MobileViTV2ForSemanticSegmentation (MobileViTV2 model)
    • SegformerConfig configuration class: SegformerForSemanticSegmentation (SegFormer model)
    • UperNetConfig configuration class: UperNetForSemanticSegmentation (UPerNet model)
  • attn_implementation (str, optional) — The attention implementation to use in the model (if relevant). Can be any of "eager" (manual implementation of the attention), "sdpa" (using F.scaled_dot_product_attention), or "flash_attention_2" (using Dao-AILab/flash-attention). By default, if available, SDPA will be used for torch>=2.1.1. The default is otherwise the manual "eager" implementation.

Instantiates one of the model classes of the library (with a semantic segmentation head) from a configuration.

Note: Loading a model from its configuration file does not load the model weights. It only affects the model’s configuration. Use from_pretrained() to load the model weights.

Examples:

>>> from transformers import AutoConfig, AutoModelForSemanticSegmentation

>>> # Download configuration from huggingface.co and cache.
>>> config = AutoConfig.from_pretrained("google-bert/bert-base-cased")
>>> model = AutoModelForSemanticSegmentation.from_config(config)

from_pretrained

< >

( *model_args **kwargs )

Parameters

  • pretrained_model_name_or_path (str or os.PathLike) — Can be either:

    • A string, the model id of a pretrained model hosted inside a model repo on huggingface.co.
    • A path to a directory containing model weights saved using save_pretrained(), e.g., ./my_model_directory/.
    • A path or url to a tensorflow index checkpoint file (e.g, ./tf_model/model.ckpt.index). In this case, from_tf should be set to True and a configuration object should be provided as config argument. This loading path is slower than converting the TensorFlow checkpoint in a PyTorch model using the provided conversion scripts and loading the PyTorch model afterwards.
  • model_args (additional positional arguments, optional) — Will be passed along to the underlying model __init__() method.
  • config (PretrainedConfig, optional) — Configuration for the model to use instead of an automatically loaded configuration. Configuration can be automatically loaded when:

    • The model is a model provided by the library (loaded with the model id string of a pretrained model).
    • The model was saved using save_pretrained() and is reloaded by supplying the save directory.
    • The model is loaded by supplying a local directory as pretrained_model_name_or_path and a configuration JSON file named config.json is found in the directory.
  • state_dict (Dict[str, torch.Tensor], optional) — A state dictionary to use instead of a state dictionary loaded from saved weights file.

    This option can be used if you want to create a model from a pretrained configuration but load your own weights. In this case though, you should check if using save_pretrained() and from_pretrained() is not a simpler option.

  • cache_dir (str or os.PathLike, optional) — Path to a directory in which a downloaded pretrained model configuration should be cached if the standard cache should not be used.
  • from_tf (bool, optional, defaults to False) — Load the model weights from a TensorFlow checkpoint save file (see docstring of pretrained_model_name_or_path argument).
  • force_download (bool, optional, defaults to False) — Whether or not to force the (re-)download of the model weights and configuration files, overriding the cached versions if they exist.
  • resume_download (bool, optional, defaults to False) — Whether or not to delete incompletely received files. Will attempt to resume the download if such a file exists.
  • proxies (Dict[str, str], optional) — A dictionary of proxy servers to use by protocol or endpoint, e.g., {'http': 'foo.bar:3128', 'http://hostname': 'foo.bar:4012'}. The proxies are used on each request.
  • output_loading_info(bool, optional, defaults to False) — Whether ot not to also return a dictionary containing missing keys, unexpected keys and error messages.
  • local_files_only(bool, optional, defaults to False) — Whether or not to only look at local files (e.g., not try downloading the model).
  • revision (str, optional, defaults to "main") — The specific model version to use. It can be a branch name, a tag name, or a commit id, since we use a git-based system for storing models and other artifacts on huggingface.co, so revision can be any identifier allowed by git.
  • trust_remote_code (bool, optional, defaults to False) — Whether or not to allow for custom models defined on the Hub in their own modeling files. This option should only be set to True for repositories you trust and in which you have read the code, as it will execute code present on the Hub on your local machine.
  • code_revision (str, optional, defaults to "main") — The specific revision to use for the code on the Hub, if the code leaves in a different repository than the rest of the model. It can be a branch name, a tag name, or a commit id, since we use a git-based system for storing models and other artifacts on huggingface.co, so revision can be any identifier allowed by git.
  • kwargs (additional keyword arguments, optional) — Can be used to update the configuration object (after it being loaded) and initiate the model (e.g., output_attentions=True). Behaves differently depending on whether a config is provided or automatically loaded:

    • If a configuration is provided with config, **kwargs will be directly passed to the underlying model’s __init__ method (we assume all relevant updates to the configuration have already been done)
    • If a configuration is not provided, kwargs will be first passed to the configuration class initialization function (from_pretrained()). Each key of kwargs that corresponds to a configuration attribute will be used to override said attribute with the supplied kwargs value. Remaining keys that do not correspond to any configuration attribute will be passed to the underlying model’s __init__ function.

Instantiate one of the model classes of the library (with a semantic segmentation head) from a pretrained model.

The model class to instantiate is selected based on the model_type property of the config object (either passed as an argument or loaded from pretrained_model_name_or_path if possible), or when it’s missing, by falling back to using pattern matching on pretrained_model_name_or_path:

  • beit — BeitForSemanticSegmentation (BEiT model)
  • data2vec-vision — Data2VecVisionForSemanticSegmentation (Data2VecVision model)
  • dpt — DPTForSemanticSegmentation (DPT model)
  • mobilenet_v2 — MobileNetV2ForSemanticSegmentation (MobileNetV2 model)
  • mobilevit — MobileViTForSemanticSegmentation (MobileViT model)
  • mobilevitv2 — MobileViTV2ForSemanticSegmentation (MobileViTV2 model)
  • segformer — SegformerForSemanticSegmentation (SegFormer model)
  • upernet — UperNetForSemanticSegmentation (UPerNet model)

The model is set in evaluation mode by default using model.eval() (so for instance, dropout modules are deactivated). To train the model, you should first set it back in training mode with model.train()

Examples:

>>> from transformers import AutoConfig, AutoModelForSemanticSegmentation

>>> # Download model and configuration from huggingface.co and cache.
>>> model = AutoModelForSemanticSegmentation.from_pretrained("google-bert/bert-base-cased")

>>> # Update configuration during loading
>>> model = AutoModelForSemanticSegmentation.from_pretrained("google-bert/bert-base-cased", output_attentions=True)
>>> model.config.output_attentions
True

>>> # Loading from a TF checkpoint file instead of a PyTorch model (slower)
>>> config = AutoConfig.from_pretrained("./tf_model/bert_tf_model_config.json")
>>> model = AutoModelForSemanticSegmentation.from_pretrained(
...     "./tf_model/bert_tf_checkpoint.ckpt.index", from_tf=True, config=config
... )

TFAutoModelForSemanticSegmentation

class transformers.TFAutoModelForSemanticSegmentation

< >

( *args **kwargs )

This is a generic model class that will be instantiated as one of the model classes of the library (with a semantic segmentation head) when created with the from_pretrained() class method or the from_config() class method.

This class cannot be instantiated directly using __init__() (throws an error).

from_config

< >

( **kwargs )

Parameters

  • config (PretrainedConfig) — The model class to instantiate is selected based on the configuration class:
  • attn_implementation (str, optional) — The attention implementation to use in the model (if relevant). Can be any of "eager" (manual implementation of the attention), "sdpa" (using F.scaled_dot_product_attention), or "flash_attention_2" (using Dao-AILab/flash-attention). By default, if available, SDPA will be used for torch>=2.1.1. The default is otherwise the manual "eager" implementation.

Instantiates one of the model classes of the library (with a semantic segmentation head) from a configuration.

Note: Loading a model from its configuration file does not load the model weights. It only affects the model’s configuration. Use from_pretrained() to load the model weights.

Examples:

>>> from transformers import AutoConfig, TFAutoModelForSemanticSegmentation

>>> # Download configuration from huggingface.co and cache.
>>> config = AutoConfig.from_pretrained("google-bert/bert-base-cased")
>>> model = TFAutoModelForSemanticSegmentation.from_config(config)

from_pretrained

< >

( *model_args **kwargs )

Parameters

  • pretrained_model_name_or_path (str or os.PathLike) — Can be either:

    • A string, the model id of a pretrained model hosted inside a model repo on huggingface.co.
    • A path to a directory containing model weights saved using save_pretrained(), e.g., ./my_model_directory/.
    • A path or url to a PyTorch state_dict save file (e.g, ./pt_model/pytorch_model.bin). In this case, from_pt should be set to True and a configuration object should be provided as config argument. This loading path is slower than converting the PyTorch model in a TensorFlow model using the provided conversion scripts and loading the TensorFlow model afterwards.
  • model_args (additional positional arguments, optional) — Will be passed along to the underlying model __init__() method.
  • config (PretrainedConfig, optional) — Configuration for the model to use instead of an automatically loaded configuration. Configuration can be automatically loaded when:

    • The model is a model provided by the library (loaded with the model id string of a pretrained model).
    • The model was saved using save_pretrained() and is reloaded by supplying the save directory.
    • The model is loaded by supplying a local directory as pretrained_model_name_or_path and a configuration JSON file named config.json is found in the directory.
  • cache_dir (str or os.PathLike, optional) — Path to a directory in which a downloaded pretrained model configuration should be cached if the standard cache should not be used.
  • from_pt (bool, optional, defaults to False) — Load the model weights from a PyTorch checkpoint save file (see docstring of pretrained_model_name_or_path argument).
  • force_download (bool, optional, defaults to False) — Whether or not to force the (re-)download of the model weights and configuration files, overriding the cached versions if they exist.
  • resume_download (bool, optional, defaults to False) — Whether or not to delete incompletely received files. Will attempt to resume the download if such a file exists.
  • proxies (Dict[str, str], optional) — A dictionary of proxy servers to use by protocol or endpoint, e.g., {'http': 'foo.bar:3128', 'http://hostname': 'foo.bar:4012'}. The proxies are used on each request.
  • output_loading_info(bool, optional, defaults to False) — Whether ot not to also return a dictionary containing missing keys, unexpected keys and error messages.
  • local_files_only(bool, optional, defaults to False) — Whether or not to only look at local files (e.g., not try downloading the model).
  • revision (str, optional, defaults to "main") — The specific model version to use. It can be a branch name, a tag name, or a commit id, since we use a git-based system for storing models and other artifacts on huggingface.co, so revision can be any identifier allowed by git.
  • trust_remote_code (bool, optional, defaults to False) — Whether or not to allow for custom models defined on the Hub in their own modeling files. This option should only be set to True for repositories you trust and in which you have read the code, as it will execute code present on the Hub on your local machine.
  • code_revision (str, optional, defaults to "main") — The specific revision to use for the code on the Hub, if the code leaves in a different repository than the rest of the model. It can be a branch name, a tag name, or a commit id, since we use a git-based system for storing models and other artifacts on huggingface.co, so revision can be any identifier allowed by git.
  • kwargs (additional keyword arguments, optional) — Can be used to update the configuration object (after it being loaded) and initiate the model (e.g., output_attentions=True). Behaves differently depending on whether a config is provided or automatically loaded:

    • If a configuration is provided with config, **kwargs will be directly passed to the underlying model’s __init__ method (we assume all relevant updates to the configuration have already been done)
    • If a configuration is not provided, kwargs will be first passed to the configuration class initialization function (from_pretrained()). Each key of kwargs that corresponds to a configuration attribute will be used to override said attribute with the supplied kwargs value. Remaining keys that do not correspond to any configuration attribute will be passed to the underlying model’s __init__ function.

Instantiate one of the model classes of the library (with a semantic segmentation head) from a pretrained model.

The model class to instantiate is selected based on the model_type property of the config object (either passed as an argument or loaded from pretrained_model_name_or_path if possible), or when it’s missing, by falling back to using pattern matching on pretrained_model_name_or_path:

  • data2vec-vision — TFData2VecVisionForSemanticSegmentation (Data2VecVision model)
  • mobilevit — TFMobileViTForSemanticSegmentation (MobileViT model)
  • segformer — TFSegformerForSemanticSegmentation (SegFormer model)

Examples:

>>> from transformers import AutoConfig, TFAutoModelForSemanticSegmentation

>>> # Download model and configuration from huggingface.co and cache.
>>> model = TFAutoModelForSemanticSegmentation.from_pretrained("google-bert/bert-base-cased")

>>> # Update configuration during loading
>>> model = TFAutoModelForSemanticSegmentation.from_pretrained("google-bert/bert-base-cased", output_attentions=True)
>>> model.config.output_attentions
True

>>> # Loading from a PyTorch checkpoint file instead of a TensorFlow model (slower)
>>> config = AutoConfig.from_pretrained("./pt_model/bert_pt_model_config.json")
>>> model = TFAutoModelForSemanticSegmentation.from_pretrained(
...     "./pt_model/bert_pytorch_model.bin", from_pt=True, config=config
... )

AutoModelForInstanceSegmentation

class transformers.AutoModelForInstanceSegmentation

< >

( *args **kwargs )

This is a generic model class that will be instantiated as one of the model classes of the library (with a instance segmentation head) when created with the from_pretrained() class method or the from_config() class method.

This class cannot be instantiated directly using __init__() (throws an error).

from_config

< >

( **kwargs )

Parameters

  • config (PretrainedConfig) — The model class to instantiate is selected based on the configuration class:

    • MaskFormerConfig configuration class: MaskFormerForInstanceSegmentation (MaskFormer model)
  • attn_implementation (str, optional) — The attention implementation to use in the model (if relevant). Can be any of "eager" (manual implementation of the attention), "sdpa" (using F.scaled_dot_product_attention), or "flash_attention_2" (using Dao-AILab/flash-attention). By default, if available, SDPA will be used for torch>=2.1.1. The default is otherwise the manual "eager" implementation.

Instantiates one of the model classes of the library (with a instance segmentation head) from a configuration.

Note: Loading a model from its configuration file does not load the model weights. It only affects the model’s configuration. Use from_pretrained() to load the model weights.

Examples:

>>> from transformers import AutoConfig, AutoModelForInstanceSegmentation

>>> # Download configuration from huggingface.co and cache.
>>> config = AutoConfig.from_pretrained("google-bert/bert-base-cased")
>>> model = AutoModelForInstanceSegmentation.from_config(config)

from_pretrained

< >

( *model_args **kwargs )

Parameters

  • pretrained_model_name_or_path (str or os.PathLike) — Can be either:

    • A string, the model id of a pretrained model hosted inside a model repo on huggingface.co.
    • A path to a directory containing model weights saved using save_pretrained(), e.g., ./my_model_directory/.
    • A path or url to a tensorflow index checkpoint file (e.g, ./tf_model/model.ckpt.index). In this case, from_tf should be set to True and a configuration object should be provided as config argument. This loading path is slower than converting the TensorFlow checkpoint in a PyTorch model using the provided conversion scripts and loading the PyTorch model afterwards.
  • model_args (additional positional arguments, optional) — Will be passed along to the underlying model __init__() method.
  • config (PretrainedConfig, optional) — Configuration for the model to use instead of an automatically loaded configuration. Configuration can be automatically loaded when:

    • The model is a model provided by the library (loaded with the model id string of a pretrained model).
    • The model was saved using save_pretrained() and is reloaded by supplying the save directory.
    • The model is loaded by supplying a local directory as pretrained_model_name_or_path and a configuration JSON file named config.json is found in the directory.
  • state_dict (Dict[str, torch.Tensor], optional) — A state dictionary to use instead of a state dictionary loaded from saved weights file.

    This option can be used if you want to create a model from a pretrained configuration but load your own weights. In this case though, you should check if using save_pretrained() and from_pretrained() is not a simpler option.

  • cache_dir (str or os.PathLike, optional) — Path to a directory in which a downloaded pretrained model configuration should be cached if the standard cache should not be used.
  • from_tf (bool, optional, defaults to False) — Load the model weights from a TensorFlow checkpoint save file (see docstring of pretrained_model_name_or_path argument).
  • force_download (bool, optional, defaults to False) — Whether or not to force the (re-)download of the model weights and configuration files, overriding the cached versions if they exist.
  • resume_download (bool, optional, defaults to False) — Whether or not to delete incompletely received files. Will attempt to resume the download if such a file exists.
  • proxies (Dict[str, str], optional) — A dictionary of proxy servers to use by protocol or endpoint, e.g., {'http': 'foo.bar:3128', 'http://hostname': 'foo.bar:4012'}. The proxies are used on each request.
  • output_loading_info(bool, optional, defaults to False) — Whether ot not to also return a dictionary containing missing keys, unexpected keys and error messages.
  • local_files_only(bool, optional, defaults to False) — Whether or not to only look at local files (e.g., not try downloading the model).
  • revision (str, optional, defaults to "main") — The specific model version to use. It can be a branch name, a tag name, or a commit id, since we use a git-based system for storing models and other artifacts on huggingface.co, so revision can be any identifier allowed by git.
  • trust_remote_code (bool, optional, defaults to False) — Whether or not to allow for custom models defined on the Hub in their own modeling files. This option should only be set to True for repositories you trust and in which you have read the code, as it will execute code present on the Hub on your local machine.
  • code_revision (str, optional, defaults to "main") — The specific revision to use for the code on the Hub, if the code leaves in a different repository than the rest of the model. It can be a branch name, a tag name, or a commit id, since we use a git-based system for storing models and other artifacts on huggingface.co, so revision can be any identifier allowed by git.
  • kwargs (additional keyword arguments, optional) — Can be used to update the configuration object (after it being loaded) and initiate the model (e.g., output_attentions=True). Behaves differently depending on whether a config is provided or automatically loaded:

    • If a configuration is provided with config, **kwargs will be directly passed to the underlying model’s __init__ method (we assume all relevant updates to the configuration have already been done)
    • If a configuration is not provided, kwargs will be first passed to the configuration class initialization function (from_pretrained()). Each key of kwargs that corresponds to a configuration attribute will be used to override said attribute with the supplied kwargs value. Remaining keys that do not correspond to any configuration attribute will be passed to the underlying model’s __init__ function.

Instantiate one of the model classes of the library (with a instance segmentation head) from a pretrained model.

The model class to instantiate is selected based on the model_type property of the config object (either passed as an argument or loaded from pretrained_model_name_or_path if possible), or when it’s missing, by falling back to using pattern matching on pretrained_model_name_or_path:

  • maskformer — MaskFormerForInstanceSegmentation (MaskFormer model)

The model is set in evaluation mode by default using model.eval() (so for instance, dropout modules are deactivated). To train the model, you should first set it back in training mode with model.train()

Examples:

>>> from transformers import AutoConfig, AutoModelForInstanceSegmentation

>>> # Download model and configuration from huggingface.co and cache.
>>> model = AutoModelForInstanceSegmentation.from_pretrained("google-bert/bert-base-cased")

>>> # Update configuration during loading
>>> model = AutoModelForInstanceSegmentation.from_pretrained("google-bert/bert-base-cased", output_attentions=True)
>>> model.config.output_attentions
True

>>> # Loading from a TF checkpoint file instead of a PyTorch model (slower)
>>> config = AutoConfig.from_pretrained("./tf_model/bert_tf_model_config.json")
>>> model = AutoModelForInstanceSegmentation.from_pretrained(
...     "./tf_model/bert_tf_checkpoint.ckpt.index", from_tf=True, config=config
... )

AutoModelForUniversalSegmentation

class transformers.AutoModelForUniversalSegmentation

< >

( *args **kwargs )

This is a generic model class that will be instantiated as one of the model classes of the library (with a universal image segmentation head) when created with the from_pretrained() class method or the from_config() class method.

This class cannot be instantiated directly using __init__() (throws an error).

from_config

< >

( **kwargs )

Parameters

  • config (PretrainedConfig) — The model class to instantiate is selected based on the configuration class:

    • DetrConfig configuration class: DetrForSegmentation (DETR model)
    • Mask2FormerConfig configuration class: Mask2FormerForUniversalSegmentation (Mask2Former model)
    • MaskFormerConfig configuration class: MaskFormerForInstanceSegmentation (MaskFormer model)
    • OneFormerConfig configuration class: OneFormerForUniversalSegmentation (OneFormer model)
  • attn_implementation (str, optional) — The attention implementation to use in the model (if relevant). Can be any of "eager" (manual implementation of the attention), "sdpa" (using F.scaled_dot_product_attention), or "flash_attention_2" (using Dao-AILab/flash-attention). By default, if available, SDPA will be used for torch>=2.1.1. The default is otherwise the manual "eager" implementation.

Instantiates one of the model classes of the library (with a universal image segmentation head) from a configuration.

Note: Loading a model from its configuration file does not load the model weights. It only affects the model’s configuration. Use from_pretrained() to load the model weights.

Examples:

>>> from transformers import AutoConfig, AutoModelForUniversalSegmentation

>>> # Download configuration from huggingface.co and cache.
>>> config = AutoConfig.from_pretrained("google-bert/bert-base-cased")
>>> model = AutoModelForUniversalSegmentation.from_config(config)

from_pretrained

< >

( *model_args **kwargs )

Parameters

  • pretrained_model_name_or_path (str or os.PathLike) — Can be either:

    • A string, the model id of a pretrained model hosted inside a model repo on huggingface.co.
    • A path to a directory containing model weights saved using save_pretrained(), e.g., ./my_model_directory/.
    • A path or url to a tensorflow index checkpoint file (e.g, ./tf_model/model.ckpt.index). In this case, from_tf should be set to True and a configuration object should be provided as config argument. This loading path is slower than converting the TensorFlow checkpoint in a PyTorch model using the provided conversion scripts and loading the PyTorch model afterwards.
  • model_args (additional positional arguments, optional) — Will be passed along to the underlying model __init__() method.
  • config (PretrainedConfig, optional) — Configuration for the model to use instead of an automatically loaded configuration. Configuration can be automatically loaded when:

    • The model is a model provided by the library (loaded with the model id string of a pretrained model).
    • The model was saved using save_pretrained() and is reloaded by supplying the save directory.
    • The model is loaded by supplying a local directory as pretrained_model_name_or_path and a configuration JSON file named config.json is found in the directory.
  • state_dict (Dict[str, torch.Tensor], optional) — A state dictionary to use instead of a state dictionary loaded from saved weights file.

    This option can be used if you want to create a model from a pretrained configuration but load your own weights. In this case though, you should check if using save_pretrained() and from_pretrained() is not a simpler option.

  • cache_dir (str or os.PathLike, optional) — Path to a directory in which a downloaded pretrained model configuration should be cached if the standard cache should not be used.
  • from_tf (bool, optional, defaults to False) — Load the model weights from a TensorFlow checkpoint save file (see docstring of pretrained_model_name_or_path argument).
  • force_download (bool, optional, defaults to False) — Whether or not to force the (re-)download of the model weights and configuration files, overriding the cached versions if they exist.
  • resume_download (bool, optional, defaults to False) — Whether or not to delete incompletely received files. Will attempt to resume the download if such a file exists.
  • proxies (Dict[str, str], optional) — A dictionary of proxy servers to use by protocol or endpoint, e.g., {'http': 'foo.bar:3128', 'http://hostname': 'foo.bar:4012'}. The proxies are used on each request.
  • output_loading_info(bool, optional, defaults to False) — Whether ot not to also return a dictionary containing missing keys, unexpected keys and error messages.
  • local_files_only(bool, optional, defaults to False) — Whether or not to only look at local files (e.g., not try downloading the model).
  • revision (str, optional, defaults to "main") — The specific model version to use. It can be a branch name, a tag name, or a commit id, since we use a git-based system for storing models and other artifacts on huggingface.co, so revision can be any identifier allowed by git.
  • trust_remote_code (bool, optional, defaults to False) — Whether or not to allow for custom models defined on the Hub in their own modeling files. This option should only be set to True for repositories you trust and in which you have read the code, as it will execute code present on the Hub on your local machine.
  • code_revision (str, optional, defaults to "main") — The specific revision to use for the code on the Hub, if the code leaves in a different repository than the rest of the model. It can be a branch name, a tag name, or a commit id, since we use a git-based system for storing models and other artifacts on huggingface.co, so revision can be any identifier allowed by git.
  • kwargs (additional keyword arguments, optional) — Can be used to update the configuration object (after it being loaded) and initiate the model (e.g., output_attentions=True). Behaves differently depending on whether a config is provided or automatically loaded:

    • If a configuration is provided with config, **kwargs will be directly passed to the underlying model’s __init__ method (we assume all relevant updates to the configuration have already been done)
    • If a configuration is not provided, kwargs will be first passed to the configuration class initialization function (from_pretrained()). Each key of kwargs that corresponds to a configuration attribute will be used to override said attribute with the supplied kwargs value. Remaining keys that do not correspond to any configuration attribute will be passed to the underlying model’s __init__ function.

Instantiate one of the model classes of the library (with a universal image segmentation head) from a pretrained model.

The model class to instantiate is selected based on the model_type property of the config object (either passed as an argument or loaded from pretrained_model_name_or_path if possible), or when it’s missing, by falling back to using pattern matching on pretrained_model_name_or_path:

  • detr — DetrForSegmentation (DETR model)
  • mask2former — Mask2FormerForUniversalSegmentation (Mask2Former model)
  • maskformer — MaskFormerForInstanceSegmentation (MaskFormer model)
  • oneformer — OneFormerForUniversalSegmentation (OneFormer model)

The model is set in evaluation mode by default using model.eval() (so for instance, dropout modules are deactivated). To train the model, you should first set it back in training mode with model.train()

Examples:

>>> from transformers import AutoConfig, AutoModelForUniversalSegmentation

>>> # Download model and configuration from huggingface.co and cache.
>>> model = AutoModelForUniversalSegmentation.from_pretrained("google-bert/bert-base-cased")

>>> # Update configuration during loading
>>> model = AutoModelForUniversalSegmentation.from_pretrained("google-bert/bert-base-cased", output_attentions=True)
>>> model.config.output_attentions
True

>>> # Loading from a TF checkpoint file instead of a PyTorch model (slower)
>>> config = AutoConfig.from_pretrained("./tf_model/bert_tf_model_config.json")
>>> model = AutoModelForUniversalSegmentation.from_pretrained(
...     "./tf_model/bert_tf_checkpoint.ckpt.index", from_tf=True, config=config
... )

AutoModelForZeroShotImageClassification

class transformers.AutoModelForZeroShotImageClassification

< >

( *args **kwargs )

This is a generic model class that will be instantiated as one of the model classes of the library (with a zero-shot image classification head) when created with the from_pretrained() class method or the from_config() class method.

This class cannot be instantiated directly using __init__() (throws an error).

from_config

< >

( **kwargs )

Parameters

Instantiates one of the model classes of the library (with a zero-shot image classification head) from a configuration.

Note: Loading a model from its configuration file does not load the model weights. It only affects the model’s configuration. Use from_pretrained() to load the model weights.

Examples:

>>> from transformers import AutoConfig, AutoModelForZeroShotImageClassification

>>> # Download configuration from huggingface.co and cache.
>>> config = AutoConfig.from_pretrained("google-bert/bert-base-cased")
>>> model = AutoModelForZeroShotImageClassification.from_config(config)

from_pretrained

< >

( *model_args **kwargs )

Parameters

  • pretrained_model_name_or_path (str or os.PathLike) — Can be either:

    • A string, the model id of a pretrained model hosted inside a model repo on huggingface.co.
    • A path to a directory containing model weights saved using save_pretrained(), e.g., ./my_model_directory/.
    • A path or url to a tensorflow index checkpoint file (e.g, ./tf_model/model.ckpt.index). In this case, from_tf should be set to True and a configuration object should be provided as config argument. This loading path is slower than converting the TensorFlow checkpoint in a PyTorch model using the provided conversion scripts and loading the PyTorch model afterwards.
  • model_args (additional positional arguments, optional) — Will be passed along to the underlying model __init__() method.
  • config (PretrainedConfig, optional) — Configuration for the model to use instead of an automatically loaded configuration. Configuration can be automatically loaded when:

    • The model is a model provided by the library (loaded with the model id string of a pretrained model).
    • The model was saved using save_pretrained() and is reloaded by supplying the save directory.
    • The model is loaded by supplying a local directory as pretrained_model_name_or_path and a configuration JSON file named config.json is found in the directory.
  • state_dict (Dict[str, torch.Tensor], optional) — A state dictionary to use instead of a state dictionary loaded from saved weights file.

    This option can be used if you want to create a model from a pretrained configuration but load your own weights. In this case though, you should check if using save_pretrained() and from_pretrained() is not a simpler option.

  • cache_dir (str or os.PathLike, optional) — Path to a directory in which a downloaded pretrained model configuration should be cached if the standard cache should not be used.
  • from_tf (bool, optional, defaults to False) — Load the model weights from a TensorFlow checkpoint save file (see docstring of pretrained_model_name_or_path argument).
  • force_download (bool, optional, defaults to False) — Whether or not to force the (re-)download of the model weights and configuration files, overriding the cached versions if they exist.
  • resume_download (bool, optional, defaults to False) — Whether or not to delete incompletely received files. Will attempt to resume the download if such a file exists.
  • proxies (Dict[str, str], optional) — A dictionary of proxy servers to use by protocol or endpoint, e.g., {'http': 'foo.bar:3128', 'http://hostname': 'foo.bar:4012'}. The proxies are used on each request.
  • output_loading_info(bool, optional, defaults to False) — Whether ot not to also return a dictionary containing missing keys, unexpected keys and error messages.
  • local_files_only(bool, optional, defaults to False) — Whether or not to only look at local files (e.g., not try downloading the model).
  • revision (str, optional, defaults to "main") — The specific model version to use. It can be a branch name, a tag name, or a commit id, since we use a git-based system for storing models and other artifacts on huggingface.co, so revision can be any identifier allowed by git.
  • trust_remote_code (bool, optional, defaults to False) — Whether or not to allow for custom models defined on the Hub in their own modeling files. This option should only be set to True for repositories you trust and in which you have read the code, as it will execute code present on the Hub on your local machine.
  • code_revision (str, optional, defaults to "main") — The specific revision to use for the code on the Hub, if the code leaves in a different repository than the rest of the model. It can be a branch name, a tag name, or a commit id, since we use a git-based system for storing models and other artifacts on huggingface.co, so revision can be any identifier allowed by git.
  • kwargs (additional keyword arguments, optional) — Can be used to update the configuration object (after it being loaded) and initiate the model (e.g., output_attentions=True). Behaves differently depending on whether a config is provided or automatically loaded:

    • If a configuration is provided with config, **kwargs will be directly passed to the underlying model’s __init__ method (we assume all relevant updates to the configuration have already been done)
    • If a configuration is not provided, kwargs will be first passed to the configuration class initialization function (from_pretrained()). Each key of kwargs that corresponds to a configuration attribute will be used to override said attribute with the supplied kwargs value. Remaining keys that do not correspond to any configuration attribute will be passed to the underlying model’s __init__ function.

Instantiate one of the model classes of the library (with a zero-shot image classification head) from a pretrained model.

The model class to instantiate is selected based on the model_type property of the config object (either passed as an argument or loaded from pretrained_model_name_or_path if possible), or when it’s missing, by falling back to using pattern matching on pretrained_model_name_or_path:

The model is set in evaluation mode by default using model.eval() (so for instance, dropout modules are deactivated). To train the model, you should first set it back in training mode with model.train()

Examples:

>>> from transformers import AutoConfig, AutoModelForZeroShotImageClassification

>>> # Download model and configuration from huggingface.co and cache.
>>> model = AutoModelForZeroShotImageClassification.from_pretrained("google-bert/bert-base-cased")

>>> # Update configuration during loading
>>> model = AutoModelForZeroShotImageClassification.from_pretrained("google-bert/bert-base-cased", output_attentions=True)
>>> model.config.output_attentions
True

>>> # Loading from a TF checkpoint file instead of a PyTorch model (slower)
>>> config = AutoConfig.from_pretrained("./tf_model/bert_tf_model_config.json")
>>> model = AutoModelForZeroShotImageClassification.from_pretrained(
...     "./tf_model/bert_tf_checkpoint.ckpt.index", from_tf=True, config=config
... )

TFAutoModelForZeroShotImageClassification

class transformers.TFAutoModelForZeroShotImageClassification

< >

( *args **kwargs )

This is a generic model class that will be instantiated as one of the model classes of the library (with a zero-shot image classification head) when created with the from_pretrained() class method or the from_config() class method.

This class cannot be instantiated directly using __init__() (throws an error).

from_config

< >

( **kwargs )

Parameters

  • config (PretrainedConfig) — The model class to instantiate is selected based on the configuration class:
  • attn_implementation (str, optional) — The attention implementation to use in the model (if relevant). Can be any of "eager" (manual implementation of the attention), "sdpa" (using F.scaled_dot_product_attention), or "flash_attention_2" (using Dao-AILab/flash-attention). By default, if available, SDPA will be used for torch>=2.1.1. The default is otherwise the manual "eager" implementation.

Instantiates one of the model classes of the library (with a zero-shot image classification head) from a configuration.

Note: Loading a model from its configuration file does not load the model weights. It only affects the model’s configuration. Use from_pretrained() to load the model weights.

Examples:

>>> from transformers import AutoConfig, TFAutoModelForZeroShotImageClassification

>>> # Download configuration from huggingface.co and cache.
>>> config = AutoConfig.from_pretrained("google-bert/bert-base-cased")
>>> model = TFAutoModelForZeroShotImageClassification.from_config(config)

from_pretrained

< >

( *model_args **kwargs )

Parameters

  • pretrained_model_name_or_path (str or os.PathLike) — Can be either:

    • A string, the model id of a pretrained model hosted inside a model repo on huggingface.co.
    • A path to a directory containing model weights saved using save_pretrained(), e.g., ./my_model_directory/.
    • A path or url to a PyTorch state_dict save file (e.g, ./pt_model/pytorch_model.bin). In this case, from_pt should be set to True and a configuration object should be provided as config argument. This loading path is slower than converting the PyTorch model in a TensorFlow model using the provided conversion scripts and loading the TensorFlow model afterwards.
  • model_args (additional positional arguments, optional) — Will be passed along to the underlying model __init__() method.
  • config (PretrainedConfig, optional) — Configuration for the model to use instead of an automatically loaded configuration. Configuration can be automatically loaded when:

    • The model is a model provided by the library (loaded with the model id string of a pretrained model).
    • The model was saved using save_pretrained() and is reloaded by supplying the save directory.
    • The model is loaded by supplying a local directory as pretrained_model_name_or_path and a configuration JSON file named config.json is found in the directory.
  • cache_dir (str or os.PathLike, optional) — Path to a directory in which a downloaded pretrained model configuration should be cached if the standard cache should not be used.
  • from_pt (bool, optional, defaults to False) — Load the model weights from a PyTorch checkpoint save file (see docstring of pretrained_model_name_or_path argument).
  • force_download (bool, optional, defaults to False) — Whether or not to force the (re-)download of the model weights and configuration files, overriding the cached versions if they exist.
  • resume_download (bool, optional, defaults to False) — Whether or not to delete incompletely received files. Will attempt to resume the download if such a file exists.
  • proxies (Dict[str, str], optional) — A dictionary of proxy servers to use by protocol or endpoint, e.g., {'http': 'foo.bar:3128', 'http://hostname': 'foo.bar:4012'}. The proxies are used on each request.
  • output_loading_info(bool, optional, defaults to False) — Whether ot not to also return a dictionary containing missing keys, unexpected keys and error messages.
  • local_files_only(bool, optional, defaults to False) — Whether or not to only look at local files (e.g., not try downloading the model).
  • revision (str, optional, defaults to "main") — The specific model version to use. It can be a branch name, a tag name, or a commit id, since we use a git-based system for storing models and other artifacts on huggingface.co, so revision can be any identifier allowed by git.
  • trust_remote_code (bool, optional, defaults to False) — Whether or not to allow for custom models defined on the Hub in their own modeling files. This option should only be set to True for repositories you trust and in which you have read the code, as it will execute code present on the Hub on your local machine.
  • code_revision (str, optional, defaults to "main") — The specific revision to use for the code on the Hub, if the code leaves in a different repository than the rest of the model. It can be a branch name, a tag name, or a commit id, since we use a git-based system for storing models and other artifacts on huggingface.co, so revision can be any identifier allowed by git.
  • kwargs (additional keyword arguments, optional) — Can be used to update the configuration object (after it being loaded) and initiate the model (e.g., output_attentions=True). Behaves differently depending on whether a config is provided or automatically loaded:

    • If a configuration is provided with config, **kwargs will be directly passed to the underlying model’s __init__ method (we assume all relevant updates to the configuration have already been done)
    • If a configuration is not provided, kwargs will be first passed to the configuration class initialization function (from_pretrained()). Each key of kwargs that corresponds to a configuration attribute will be used to override said attribute with the supplied kwargs value. Remaining keys that do not correspond to any configuration attribute will be passed to the underlying model’s __init__ function.

Instantiate one of the model classes of the library (with a zero-shot image classification head) from a pretrained model.

The model class to instantiate is selected based on the model_type property of the config object (either passed as an argument or loaded from pretrained_model_name_or_path if possible), or when it’s missing, by falling back to using pattern matching on pretrained_model_name_or_path:

Examples:

>>> from transformers import AutoConfig, TFAutoModelForZeroShotImageClassification

>>> # Download model and configuration from huggingface.co and cache.
>>> model = TFAutoModelForZeroShotImageClassification.from_pretrained("google-bert/bert-base-cased")

>>> # Update configuration during loading
>>> model = TFAutoModelForZeroShotImageClassification.from_pretrained("google-bert/bert-base-cased", output_attentions=True)
>>> model.config.output_attentions
True

>>> # Loading from a PyTorch checkpoint file instead of a TensorFlow model (slower)
>>> config = AutoConfig.from_pretrained("./pt_model/bert_pt_model_config.json")
>>> model = TFAutoModelForZeroShotImageClassification.from_pretrained(
...     "./pt_model/bert_pytorch_model.bin", from_pt=True, config=config
... )

AutoModelForZeroShotObjectDetection

class transformers.AutoModelForZeroShotObjectDetection

< >

( *args **kwargs )

This is a generic model class that will be instantiated as one of the model classes of the library (with a zero-shot object detection head) when created with the from_pretrained() class method or the from_config() class method.

This class cannot be instantiated directly using __init__() (throws an error).

from_config

< >

( **kwargs )

Parameters

  • config (PretrainedConfig) — The model class to instantiate is selected based on the configuration class:

    • GroundingDinoConfig configuration class: GroundingDinoForObjectDetection (Grounding DINO model)
    • OwlViTConfig configuration class: OwlViTForObjectDetection (OWL-ViT model)
    • Owlv2Config configuration class: Owlv2ForObjectDetection (OWLv2 model)
  • attn_implementation (str, optional) — The attention implementation to use in the model (if relevant). Can be any of "eager" (manual implementation of the attention), "sdpa" (using F.scaled_dot_product_attention), or "flash_attention_2" (using Dao-AILab/flash-attention). By default, if available, SDPA will be used for torch>=2.1.1. The default is otherwise the manual "eager" implementation.

Instantiates one of the model classes of the library (with a zero-shot object detection head) from a configuration.

Note: Loading a model from its configuration file does not load the model weights. It only affects the model’s configuration. Use from_pretrained() to load the model weights.

Examples:

>>> from transformers import AutoConfig, AutoModelForZeroShotObjectDetection

>>> # Download configuration from huggingface.co and cache.
>>> config = AutoConfig.from_pretrained("google-bert/bert-base-cased")
>>> model = AutoModelForZeroShotObjectDetection.from_config(config)

from_pretrained

< >

( *model_args **kwargs )

Parameters

  • pretrained_model_name_or_path (str or os.PathLike) — Can be either:

    • A string, the model id of a pretrained model hosted inside a model repo on huggingface.co.
    • A path to a directory containing model weights saved using save_pretrained(), e.g., ./my_model_directory/.
    • A path or url to a tensorflow index checkpoint file (e.g, ./tf_model/model.ckpt.index). In this case, from_tf should be set to True and a configuration object should be provided as config argument. This loading path is slower than converting the TensorFlow checkpoint in a PyTorch model using the provided conversion scripts and loading the PyTorch model afterwards.
  • model_args (additional positional arguments, optional) — Will be passed along to the underlying model __init__() method.
  • config (PretrainedConfig, optional) — Configuration for the model to use instead of an automatically loaded configuration. Configuration can be automatically loaded when:

    • The model is a model provided by the library (loaded with the model id string of a pretrained model).
    • The model was saved using save_pretrained() and is reloaded by supplying the save directory.
    • The model is loaded by supplying a local directory as pretrained_model_name_or_path and a configuration JSON file named config.json is found in the directory.
  • state_dict (Dict[str, torch.Tensor], optional) — A state dictionary to use instead of a state dictionary loaded from saved weights file.

    This option can be used if you want to create a model from a pretrained configuration but load your own weights. In this case though, you should check if using save_pretrained() and from_pretrained() is not a simpler option.

  • cache_dir (str or os.PathLike, optional) — Path to a directory in which a downloaded pretrained model configuration should be cached if the standard cache should not be used.
  • from_tf (bool, optional, defaults to False) — Load the model weights from a TensorFlow checkpoint save file (see docstring of pretrained_model_name_or_path argument).
  • force_download (bool, optional, defaults to False) — Whether or not to force the (re-)download of the model weights and configuration files, overriding the cached versions if they exist.
  • resume_download (bool, optional, defaults to False) — Whether or not to delete incompletely received files. Will attempt to resume the download if such a file exists.
  • proxies (Dict[str, str], optional) — A dictionary of proxy servers to use by protocol or endpoint, e.g., {'http': 'foo.bar:3128', 'http://hostname': 'foo.bar:4012'}. The proxies are used on each request.
  • output_loading_info(bool, optional, defaults to False) — Whether ot not to also return a dictionary containing missing keys, unexpected keys and error messages.
  • local_files_only(bool, optional, defaults to False) — Whether or not to only look at local files (e.g., not try downloading the model).
  • revision (str, optional, defaults to "main") — The specific model version to use. It can be a branch name, a tag name, or a commit id, since we use a git-based system for storing models and other artifacts on huggingface.co, so revision can be any identifier allowed by git.
  • trust_remote_code (bool, optional, defaults to False) — Whether or not to allow for custom models defined on the Hub in their own modeling files. This option should only be set to True for repositories you trust and in which you have read the code, as it will execute code present on the Hub on your local machine.
  • code_revision (str, optional, defaults to "main") — The specific revision to use for the code on the Hub, if the code leaves in a different repository than the rest of the model. It can be a branch name, a tag name, or a commit id, since we use a git-based system for storing models and other artifacts on huggingface.co, so revision can be any identifier allowed by git.
  • kwargs (additional keyword arguments, optional) — Can be used to update the configuration object (after it being loaded) and initiate the model (e.g., output_attentions=True). Behaves differently depending on whether a config is provided or automatically loaded:

    • If a configuration is provided with config, **kwargs will be directly passed to the underlying model’s __init__ method (we assume all relevant updates to the configuration have already been done)
    • If a configuration is not provided, kwargs will be first passed to the configuration class initialization function (from_pretrained()). Each key of kwargs that corresponds to a configuration attribute will be used to override said attribute with the supplied kwargs value. Remaining keys that do not correspond to any configuration attribute will be passed to the underlying model’s __init__ function.

Instantiate one of the model classes of the library (with a zero-shot object detection head) from a pretrained model.

The model class to instantiate is selected based on the model_type property of the config object (either passed as an argument or loaded from pretrained_model_name_or_path if possible), or when it’s missing, by falling back to using pattern matching on pretrained_model_name_or_path:

  • grounding-dino — GroundingDinoForObjectDetection (Grounding DINO model)
  • owlv2 — Owlv2ForObjectDetection (OWLv2 model)
  • owlvit — OwlViTForObjectDetection (OWL-ViT model)

The model is set in evaluation mode by default using model.eval() (so for instance, dropout modules are deactivated). To train the model, you should first set it back in training mode with model.train()

Examples:

>>> from transformers import AutoConfig, AutoModelForZeroShotObjectDetection

>>> # Download model and configuration from huggingface.co and cache.
>>> model = AutoModelForZeroShotObjectDetection.from_pretrained("google-bert/bert-base-cased")

>>> # Update configuration during loading
>>> model = AutoModelForZeroShotObjectDetection.from_pretrained("google-bert/bert-base-cased", output_attentions=True)
>>> model.config.output_attentions
True

>>> # Loading from a TF checkpoint file instead of a PyTorch model (slower)
>>> config = AutoConfig.from_pretrained("./tf_model/bert_tf_model_config.json")
>>> model = AutoModelForZeroShotObjectDetection.from_pretrained(
...     "./tf_model/bert_tf_checkpoint.ckpt.index", from_tf=True, config=config
... )

Audio

以下の自動クラスは、次の音声タスクに利用可能です。

AutoModelForAudioClassification

class transformers.AutoModelForAudioClassification

< >

( *args **kwargs )

This is a generic model class that will be instantiated as one of the model classes of the library (with a audio classification head) when created with the from_pretrained() class method or the from_config() class method.

This class cannot be instantiated directly using __init__() (throws an error).

from_config

< >

( **kwargs )

Parameters

  • config (PretrainedConfig) — The model class to instantiate is selected based on the configuration class:

    • ASTConfig configuration class: ASTForAudioClassification (Audio Spectrogram Transformer model)
    • Data2VecAudioConfig configuration class: Data2VecAudioForSequenceClassification (Data2VecAudio model)
    • HubertConfig configuration class: HubertForSequenceClassification (Hubert model)
    • SEWConfig configuration class: SEWForSequenceClassification (SEW model)
    • SEWDConfig configuration class: SEWDForSequenceClassification (SEW-D model)
    • UniSpeechConfig configuration class: UniSpeechForSequenceClassification (UniSpeech model)
    • UniSpeechSatConfig configuration class: UniSpeechSatForSequenceClassification (UniSpeechSat model)
    • Wav2Vec2BertConfig configuration class: Wav2Vec2BertForSequenceClassification (Wav2Vec2-BERT model)
    • Wav2Vec2Config configuration class: Wav2Vec2ForSequenceClassification (Wav2Vec2 model)
    • Wav2Vec2ConformerConfig configuration class: Wav2Vec2ConformerForSequenceClassification (Wav2Vec2-Conformer model)
    • WavLMConfig configuration class: WavLMForSequenceClassification (WavLM model)
    • WhisperConfig configuration class: WhisperForAudioClassification (Whisper model)
  • attn_implementation (str, optional) — The attention implementation to use in the model (if relevant). Can be any of "eager" (manual implementation of the attention), "sdpa" (using F.scaled_dot_product_attention), or "flash_attention_2" (using Dao-AILab/flash-attention). By default, if available, SDPA will be used for torch>=2.1.1. The default is otherwise the manual "eager" implementation.

Instantiates one of the model classes of the library (with a audio classification head) from a configuration.

Note: Loading a model from its configuration file does not load the model weights. It only affects the model’s configuration. Use from_pretrained() to load the model weights.

Examples:

>>> from transformers import AutoConfig, AutoModelForAudioClassification

>>> # Download configuration from huggingface.co and cache.
>>> config = AutoConfig.from_pretrained("google-bert/bert-base-cased")
>>> model = AutoModelForAudioClassification.from_config(config)

from_pretrained

< >

( *model_args **kwargs )

Parameters

  • pretrained_model_name_or_path (str or os.PathLike) — Can be either:

    • A string, the model id of a pretrained model hosted inside a model repo on huggingface.co.
    • A path to a directory containing model weights saved using save_pretrained(), e.g., ./my_model_directory/.
    • A path or url to a tensorflow index checkpoint file (e.g, ./tf_model/model.ckpt.index). In this case, from_tf should be set to True and a configuration object should be provided as config argument. This loading path is slower than converting the TensorFlow checkpoint in a PyTorch model using the provided conversion scripts and loading the PyTorch model afterwards.
  • model_args (additional positional arguments, optional) — Will be passed along to the underlying model __init__() method.
  • config (PretrainedConfig, optional) — Configuration for the model to use instead of an automatically loaded configuration. Configuration can be automatically loaded when:

    • The model is a model provided by the library (loaded with the model id string of a pretrained model).
    • The model was saved using save_pretrained() and is reloaded by supplying the save directory.
    • The model is loaded by supplying a local directory as pretrained_model_name_or_path and a configuration JSON file named config.json is found in the directory.
  • state_dict (Dict[str, torch.Tensor], optional) — A state dictionary to use instead of a state dictionary loaded from saved weights file.

    This option can be used if you want to create a model from a pretrained configuration but load your own weights. In this case though, you should check if using save_pretrained() and from_pretrained() is not a simpler option.

  • cache_dir (str or os.PathLike, optional) — Path to a directory in which a downloaded pretrained model configuration should be cached if the standard cache should not be used.
  • from_tf (bool, optional, defaults to False) — Load the model weights from a TensorFlow checkpoint save file (see docstring of pretrained_model_name_or_path argument).
  • force_download (bool, optional, defaults to False) — Whether or not to force the (re-)download of the model weights and configuration files, overriding the cached versions if they exist.
  • resume_download (bool, optional, defaults to False) — Whether or not to delete incompletely received files. Will attempt to resume the download if such a file exists.
  • proxies (Dict[str, str], optional) — A dictionary of proxy servers to use by protocol or endpoint, e.g., {'http': 'foo.bar:3128', 'http://hostname': 'foo.bar:4012'}. The proxies are used on each request.
  • output_loading_info(bool, optional, defaults to False) — Whether ot not to also return a dictionary containing missing keys, unexpected keys and error messages.
  • local_files_only(bool, optional, defaults to False) — Whether or not to only look at local files (e.g., not try downloading the model).
  • revision (str, optional, defaults to "main") — The specific model version to use. It can be a branch name, a tag name, or a commit id, since we use a git-based system for storing models and other artifacts on huggingface.co, so revision can be any identifier allowed by git.
  • trust_remote_code (bool, optional, defaults to False) — Whether or not to allow for custom models defined on the Hub in their own modeling files. This option should only be set to True for repositories you trust and in which you have read the code, as it will execute code present on the Hub on your local machine.
  • code_revision (str, optional, defaults to "main") — The specific revision to use for the code on the Hub, if the code leaves in a different repository than the rest of the model. It can be a branch name, a tag name, or a commit id, since we use a git-based system for storing models and other artifacts on huggingface.co, so revision can be any identifier allowed by git.
  • kwargs (additional keyword arguments, optional) — Can be used to update the configuration object (after it being loaded) and initiate the model (e.g., output_attentions=True). Behaves differently depending on whether a config is provided or automatically loaded:

    • If a configuration is provided with config, **kwargs will be directly passed to the underlying model’s __init__ method (we assume all relevant updates to the configuration have already been done)
    • If a configuration is not provided, kwargs will be first passed to the configuration class initialization function (from_pretrained()). Each key of kwargs that corresponds to a configuration attribute will be used to override said attribute with the supplied kwargs value. Remaining keys that do not correspond to any configuration attribute will be passed to the underlying model’s __init__ function.

Instantiate one of the model classes of the library (with a audio classification head) from a pretrained model.

The model class to instantiate is selected based on the model_type property of the config object (either passed as an argument or loaded from pretrained_model_name_or_path if possible), or when it’s missing, by falling back to using pattern matching on pretrained_model_name_or_path:

  • audio-spectrogram-transformer — ASTForAudioClassification (Audio Spectrogram Transformer model)
  • data2vec-audio — Data2VecAudioForSequenceClassification (Data2VecAudio model)
  • hubert — HubertForSequenceClassification (Hubert model)
  • sew — SEWForSequenceClassification (SEW model)
  • sew-d — SEWDForSequenceClassification (SEW-D model)
  • unispeech — UniSpeechForSequenceClassification (UniSpeech model)
  • unispeech-sat — UniSpeechSatForSequenceClassification (UniSpeechSat model)
  • wav2vec2 — Wav2Vec2ForSequenceClassification (Wav2Vec2 model)
  • wav2vec2-bert — Wav2Vec2BertForSequenceClassification (Wav2Vec2-BERT model)
  • wav2vec2-conformer — Wav2Vec2ConformerForSequenceClassification (Wav2Vec2-Conformer model)
  • wavlm — WavLMForSequenceClassification (WavLM model)
  • whisper — WhisperForAudioClassification (Whisper model)

The model is set in evaluation mode by default using model.eval() (so for instance, dropout modules are deactivated). To train the model, you should first set it back in training mode with model.train()

Examples:

>>> from transformers import AutoConfig, AutoModelForAudioClassification

>>> # Download model and configuration from huggingface.co and cache.
>>> model = AutoModelForAudioClassification.from_pretrained("google-bert/bert-base-cased")

>>> # Update configuration during loading
>>> model = AutoModelForAudioClassification.from_pretrained("google-bert/bert-base-cased", output_attentions=True)
>>> model.config.output_attentions
True

>>> # Loading from a TF checkpoint file instead of a PyTorch model (slower)
>>> config = AutoConfig.from_pretrained("./tf_model/bert_tf_model_config.json")
>>> model = AutoModelForAudioClassification.from_pretrained(
...     "./tf_model/bert_tf_checkpoint.ckpt.index", from_tf=True, config=config
... )

AutoModelForAudioFrameClassification

class transformers.TFAutoModelForAudioClassification

< >

( *args **kwargs )

This is a generic model class that will be instantiated as one of the model classes of the library (with a audio classification head) when created with the from_pretrained() class method or the from_config() class method.

This class cannot be instantiated directly using __init__() (throws an error).

from_config

< >

( **kwargs )

Parameters

  • config (PretrainedConfig) — The model class to instantiate is selected based on the configuration class:

    • Wav2Vec2Config configuration class: TFWav2Vec2ForSequenceClassification (Wav2Vec2 model)
  • attn_implementation (str, optional) — The attention implementation to use in the model (if relevant). Can be any of "eager" (manual implementation of the attention), "sdpa" (using F.scaled_dot_product_attention), or "flash_attention_2" (using Dao-AILab/flash-attention). By default, if available, SDPA will be used for torch>=2.1.1. The default is otherwise the manual "eager" implementation.

Instantiates one of the model classes of the library (with a audio classification head) from a configuration.

Note: Loading a model from its configuration file does not load the model weights. It only affects the model’s configuration. Use from_pretrained() to load the model weights.

Examples:

>>> from transformers import AutoConfig, TFAutoModelForAudioClassification

>>> # Download configuration from huggingface.co and cache.
>>> config = AutoConfig.from_pretrained("google-bert/bert-base-cased")
>>> model = TFAutoModelForAudioClassification.from_config(config)

from_pretrained

< >

( *model_args **kwargs )

Parameters

  • pretrained_model_name_or_path (str or os.PathLike) — Can be either:

    • A string, the model id of a pretrained model hosted inside a model repo on huggingface.co.
    • A path to a directory containing model weights saved using save_pretrained(), e.g., ./my_model_directory/.
    • A path or url to a PyTorch state_dict save file (e.g, ./pt_model/pytorch_model.bin). In this case, from_pt should be set to True and a configuration object should be provided as config argument. This loading path is slower than converting the PyTorch model in a TensorFlow model using the provided conversion scripts and loading the TensorFlow model afterwards.
  • model_args (additional positional arguments, optional) — Will be passed along to the underlying model __init__() method.
  • config (PretrainedConfig, optional) — Configuration for the model to use instead of an automatically loaded configuration. Configuration can be automatically loaded when:

    • The model is a model provided by the library (loaded with the model id string of a pretrained model).
    • The model was saved using save_pretrained() and is reloaded by supplying the save directory.
    • The model is loaded by supplying a local directory as pretrained_model_name_or_path and a configuration JSON file named config.json is found in the directory.
  • cache_dir (str or os.PathLike, optional) — Path to a directory in which a downloaded pretrained model configuration should be cached if the standard cache should not be used.
  • from_pt (bool, optional, defaults to False) — Load the model weights from a PyTorch checkpoint save file (see docstring of pretrained_model_name_or_path argument).
  • force_download (bool, optional, defaults to False) — Whether or not to force the (re-)download of the model weights and configuration files, overriding the cached versions if they exist.
  • resume_download (bool, optional, defaults to False) — Whether or not to delete incompletely received files. Will attempt to resume the download if such a file exists.
  • proxies (Dict[str, str], optional) — A dictionary of proxy servers to use by protocol or endpoint, e.g., {'http': 'foo.bar:3128', 'http://hostname': 'foo.bar:4012'}. The proxies are used on each request.
  • output_loading_info(bool, optional, defaults to False) — Whether ot not to also return a dictionary containing missing keys, unexpected keys and error messages.
  • local_files_only(bool, optional, defaults to False) — Whether or not to only look at local files (e.g., not try downloading the model).
  • revision (str, optional, defaults to "main") — The specific model version to use. It can be a branch name, a tag name, or a commit id, since we use a git-based system for storing models and other artifacts on huggingface.co, so revision can be any identifier allowed by git.
  • trust_remote_code (bool, optional, defaults to False) — Whether or not to allow for custom models defined on the Hub in their own modeling files. This option should only be set to True for repositories you trust and in which you have read the code, as it will execute code present on the Hub on your local machine.
  • code_revision (str, optional, defaults to "main") — The specific revision to use for the code on the Hub, if the code leaves in a different repository than the rest of the model. It can be a branch name, a tag name, or a commit id, since we use a git-based system for storing models and other artifacts on huggingface.co, so revision can be any identifier allowed by git.
  • kwargs (additional keyword arguments, optional) — Can be used to update the configuration object (after it being loaded) and initiate the model (e.g., output_attentions=True). Behaves differently depending on whether a config is provided or automatically loaded:

    • If a configuration is provided with config, **kwargs will be directly passed to the underlying model’s __init__ method (we assume all relevant updates to the configuration have already been done)
    • If a configuration is not provided, kwargs will be first passed to the configuration class initialization function (from_pretrained()). Each key of kwargs that corresponds to a configuration attribute will be used to override said attribute with the supplied kwargs value. Remaining keys that do not correspond to any configuration attribute will be passed to the underlying model’s __init__ function.

Instantiate one of the model classes of the library (with a audio classification head) from a pretrained model.

The model class to instantiate is selected based on the model_type property of the config object (either passed as an argument or loaded from pretrained_model_name_or_path if possible), or when it’s missing, by falling back to using pattern matching on pretrained_model_name_or_path:

  • wav2vec2 — TFWav2Vec2ForSequenceClassification (Wav2Vec2 model)

Examples:

>>> from transformers import AutoConfig, TFAutoModelForAudioClassification

>>> # Download model and configuration from huggingface.co and cache.
>>> model = TFAutoModelForAudioClassification.from_pretrained("google-bert/bert-base-cased")

>>> # Update configuration during loading
>>> model = TFAutoModelForAudioClassification.from_pretrained("google-bert/bert-base-cased", output_attentions=True)
>>> model.config.output_attentions
True

>>> # Loading from a PyTorch checkpoint file instead of a TensorFlow model (slower)
>>> config = AutoConfig.from_pretrained("./pt_model/bert_pt_model_config.json")
>>> model = TFAutoModelForAudioClassification.from_pretrained(
...     "./pt_model/bert_pytorch_model.bin", from_pt=True, config=config
... )

TFAutoModelForAudioFrameClassification

class transformers.AutoModelForAudioFrameClassification

< >

( *args **kwargs )

This is a generic model class that will be instantiated as one of the model classes of the library (with a audio frame (token) classification head) when created with the from_pretrained() class method or the from_config() class method.

This class cannot be instantiated directly using __init__() (throws an error).

from_config

< >

( **kwargs )

Parameters

  • config (PretrainedConfig) — The model class to instantiate is selected based on the configuration class:

    • Data2VecAudioConfig configuration class: Data2VecAudioForAudioFrameClassification (Data2VecAudio model)
    • UniSpeechSatConfig configuration class: UniSpeechSatForAudioFrameClassification (UniSpeechSat model)
    • Wav2Vec2BertConfig configuration class: Wav2Vec2BertForAudioFrameClassification (Wav2Vec2-BERT model)
    • Wav2Vec2Config configuration class: Wav2Vec2ForAudioFrameClassification (Wav2Vec2 model)
    • Wav2Vec2ConformerConfig configuration class: Wav2Vec2ConformerForAudioFrameClassification (Wav2Vec2-Conformer model)
    • WavLMConfig configuration class: WavLMForAudioFrameClassification (WavLM model)
  • attn_implementation (str, optional) — The attention implementation to use in the model (if relevant). Can be any of "eager" (manual implementation of the attention), "sdpa" (using F.scaled_dot_product_attention), or "flash_attention_2" (using Dao-AILab/flash-attention). By default, if available, SDPA will be used for torch>=2.1.1. The default is otherwise the manual "eager" implementation.

Instantiates one of the model classes of the library (with a audio frame (token) classification head) from a configuration.

Note: Loading a model from its configuration file does not load the model weights. It only affects the model’s configuration. Use from_pretrained() to load the model weights.

Examples:

>>> from transformers import AutoConfig, AutoModelForAudioFrameClassification

>>> # Download configuration from huggingface.co and cache.
>>> config = AutoConfig.from_pretrained("google-bert/bert-base-cased")
>>> model = AutoModelForAudioFrameClassification.from_config(config)

from_pretrained

< >

( *model_args **kwargs )

Parameters

  • pretrained_model_name_or_path (str or os.PathLike) — Can be either:

    • A string, the model id of a pretrained model hosted inside a model repo on huggingface.co.
    • A path to a directory containing model weights saved using save_pretrained(), e.g., ./my_model_directory/.
    • A path or url to a tensorflow index checkpoint file (e.g, ./tf_model/model.ckpt.index). In this case, from_tf should be set to True and a configuration object should be provided as config argument. This loading path is slower than converting the TensorFlow checkpoint in a PyTorch model using the provided conversion scripts and loading the PyTorch model afterwards.
  • model_args (additional positional arguments, optional) — Will be passed along to the underlying model __init__() method.
  • config (PretrainedConfig, optional) — Configuration for the model to use instead of an automatically loaded configuration. Configuration can be automatically loaded when:

    • The model is a model provided by the library (loaded with the model id string of a pretrained model).
    • The model was saved using save_pretrained() and is reloaded by supplying the save directory.
    • The model is loaded by supplying a local directory as pretrained_model_name_or_path and a configuration JSON file named config.json is found in the directory.
  • state_dict (Dict[str, torch.Tensor], optional) — A state dictionary to use instead of a state dictionary loaded from saved weights file.

    This option can be used if you want to create a model from a pretrained configuration but load your own weights. In this case though, you should check if using save_pretrained() and from_pretrained() is not a simpler option.

  • cache_dir (str or os.PathLike, optional) — Path to a directory in which a downloaded pretrained model configuration should be cached if the standard cache should not be used.
  • from_tf (bool, optional, defaults to False) — Load the model weights from a TensorFlow checkpoint save file (see docstring of pretrained_model_name_or_path argument).
  • force_download (bool, optional, defaults to False) — Whether or not to force the (re-)download of the model weights and configuration files, overriding the cached versions if they exist.
  • resume_download (bool, optional, defaults to False) — Whether or not to delete incompletely received files. Will attempt to resume the download if such a file exists.
  • proxies (Dict[str, str], optional) — A dictionary of proxy servers to use by protocol or endpoint, e.g., {'http': 'foo.bar:3128', 'http://hostname': 'foo.bar:4012'}. The proxies are used on each request.
  • output_loading_info(bool, optional, defaults to False) — Whether ot not to also return a dictionary containing missing keys, unexpected keys and error messages.
  • local_files_only(bool, optional, defaults to False) — Whether or not to only look at local files (e.g., not try downloading the model).
  • revision (str, optional, defaults to "main") — The specific model version to use. It can be a branch name, a tag name, or a commit id, since we use a git-based system for storing models and other artifacts on huggingface.co, so revision can be any identifier allowed by git.
  • trust_remote_code (bool, optional, defaults to False) — Whether or not to allow for custom models defined on the Hub in their own modeling files. This option should only be set to True for repositories you trust and in which you have read the code, as it will execute code present on the Hub on your local machine.
  • code_revision (str, optional, defaults to "main") — The specific revision to use for the code on the Hub, if the code leaves in a different repository than the rest of the model. It can be a branch name, a tag name, or a commit id, since we use a git-based system for storing models and other artifacts on huggingface.co, so revision can be any identifier allowed by git.
  • kwargs (additional keyword arguments, optional) — Can be used to update the configuration object (after it being loaded) and initiate the model (e.g., output_attentions=True). Behaves differently depending on whether a config is provided or automatically loaded:

    • If a configuration is provided with config, **kwargs will be directly passed to the underlying model’s __init__ method (we assume all relevant updates to the configuration have already been done)
    • If a configuration is not provided, kwargs will be first passed to the configuration class initialization function (from_pretrained()). Each key of kwargs that corresponds to a configuration attribute will be used to override said attribute with the supplied kwargs value. Remaining keys that do not correspond to any configuration attribute will be passed to the underlying model’s __init__ function.

Instantiate one of the model classes of the library (with a audio frame (token) classification head) from a pretrained model.

The model class to instantiate is selected based on the model_type property of the config object (either passed as an argument or loaded from pretrained_model_name_or_path if possible), or when it’s missing, by falling back to using pattern matching on pretrained_model_name_or_path:

  • data2vec-audio — Data2VecAudioForAudioFrameClassification (Data2VecAudio model)
  • unispeech-sat — UniSpeechSatForAudioFrameClassification (UniSpeechSat model)
  • wav2vec2 — Wav2Vec2ForAudioFrameClassification (Wav2Vec2 model)
  • wav2vec2-bert — Wav2Vec2BertForAudioFrameClassification (Wav2Vec2-BERT model)
  • wav2vec2-conformer — Wav2Vec2ConformerForAudioFrameClassification (Wav2Vec2-Conformer model)
  • wavlm — WavLMForAudioFrameClassification (WavLM model)

The model is set in evaluation mode by default using model.eval() (so for instance, dropout modules are deactivated). To train the model, you should first set it back in training mode with model.train()

Examples:

>>> from transformers import AutoConfig, AutoModelForAudioFrameClassification

>>> # Download model and configuration from huggingface.co and cache.
>>> model = AutoModelForAudioFrameClassification.from_pretrained("google-bert/bert-base-cased")

>>> # Update configuration during loading
>>> model = AutoModelForAudioFrameClassification.from_pretrained("google-bert/bert-base-cased", output_attentions=True)
>>> model.config.output_attentions
True

>>> # Loading from a TF checkpoint file instead of a PyTorch model (slower)
>>> config = AutoConfig.from_pretrained("./tf_model/bert_tf_model_config.json")
>>> model = AutoModelForAudioFrameClassification.from_pretrained(
...     "./tf_model/bert_tf_checkpoint.ckpt.index", from_tf=True, config=config
... )

AutoModelForCTC

class transformers.AutoModelForCTC

< >

( *args **kwargs )

This is a generic model class that will be instantiated as one of the model classes of the library (with a connectionist temporal classification head) when created with the from_pretrained() class method or the from_config() class method.

This class cannot be instantiated directly using __init__() (throws an error).

from_config

< >

( **kwargs )

Parameters

  • config (PretrainedConfig) — The model class to instantiate is selected based on the configuration class:

    • Data2VecAudioConfig configuration class: Data2VecAudioForCTC (Data2VecAudio model)
    • HubertConfig configuration class: HubertForCTC (Hubert model)
    • MCTCTConfig configuration class: MCTCTForCTC (M-CTC-T model)
    • SEWConfig configuration class: SEWForCTC (SEW model)
    • SEWDConfig configuration class: SEWDForCTC (SEW-D model)
    • UniSpeechConfig configuration class: UniSpeechForCTC (UniSpeech model)
    • UniSpeechSatConfig configuration class: UniSpeechSatForCTC (UniSpeechSat model)
    • Wav2Vec2BertConfig configuration class: Wav2Vec2BertForCTC (Wav2Vec2-BERT model)
    • Wav2Vec2Config configuration class: Wav2Vec2ForCTC (Wav2Vec2 model)
    • Wav2Vec2ConformerConfig configuration class: Wav2Vec2ConformerForCTC (Wav2Vec2-Conformer model)
    • WavLMConfig configuration class: WavLMForCTC (WavLM model)
  • attn_implementation (str, optional) — The attention implementation to use in the model (if relevant). Can be any of "eager" (manual implementation of the attention), "sdpa" (using F.scaled_dot_product_attention), or "flash_attention_2" (using Dao-AILab/flash-attention). By default, if available, SDPA will be used for torch>=2.1.1. The default is otherwise the manual "eager" implementation.

Instantiates one of the model classes of the library (with a connectionist temporal classification head) from a configuration.

Note: Loading a model from its configuration file does not load the model weights. It only affects the model’s configuration. Use from_pretrained() to load the model weights.

Examples:

>>> from transformers import AutoConfig, AutoModelForCTC

>>> # Download configuration from huggingface.co and cache.
>>> config = AutoConfig.from_pretrained("google-bert/bert-base-cased")
>>> model = AutoModelForCTC.from_config(config)

from_pretrained

< >

( *model_args **kwargs )

Parameters

  • pretrained_model_name_or_path (str or os.PathLike) — Can be either:

    • A string, the model id of a pretrained model hosted inside a model repo on huggingface.co.
    • A path to a directory containing model weights saved using save_pretrained(), e.g., ./my_model_directory/.
    • A path or url to a tensorflow index checkpoint file (e.g, ./tf_model/model.ckpt.index). In this case, from_tf should be set to True and a configuration object should be provided as config argument. This loading path is slower than converting the TensorFlow checkpoint in a PyTorch model using the provided conversion scripts and loading the PyTorch model afterwards.
  • model_args (additional positional arguments, optional) — Will be passed along to the underlying model __init__() method.
  • config (PretrainedConfig, optional) — Configuration for the model to use instead of an automatically loaded configuration. Configuration can be automatically loaded when:

    • The model is a model provided by the library (loaded with the model id string of a pretrained model).
    • The model was saved using save_pretrained() and is reloaded by supplying the save directory.
    • The model is loaded by supplying a local directory as pretrained_model_name_or_path and a configuration JSON file named config.json is found in the directory.
  • state_dict (Dict[str, torch.Tensor], optional) — A state dictionary to use instead of a state dictionary loaded from saved weights file.

    This option can be used if you want to create a model from a pretrained configuration but load your own weights. In this case though, you should check if using save_pretrained() and from_pretrained() is not a simpler option.

  • cache_dir (str or os.PathLike, optional) — Path to a directory in which a downloaded pretrained model configuration should be cached if the standard cache should not be used.
  • from_tf (bool, optional, defaults to False) — Load the model weights from a TensorFlow checkpoint save file (see docstring of pretrained_model_name_or_path argument).
  • force_download (bool, optional, defaults to False) — Whether or not to force the (re-)download of the model weights and configuration files, overriding the cached versions if they exist.
  • resume_download (bool, optional, defaults to False) — Whether or not to delete incompletely received files. Will attempt to resume the download if such a file exists.
  • proxies (Dict[str, str], optional) — A dictionary of proxy servers to use by protocol or endpoint, e.g., {'http': 'foo.bar:3128', 'http://hostname': 'foo.bar:4012'}. The proxies are used on each request.
  • output_loading_info(bool, optional, defaults to False) — Whether ot not to also return a dictionary containing missing keys, unexpected keys and error messages.
  • local_files_only(bool, optional, defaults to False) — Whether or not to only look at local files (e.g., not try downloading the model).
  • revision (str, optional, defaults to "main") — The specific model version to use. It can be a branch name, a tag name, or a commit id, since we use a git-based system for storing models and other artifacts on huggingface.co, so revision can be any identifier allowed by git.
  • trust_remote_code (bool, optional, defaults to False) — Whether or not to allow for custom models defined on the Hub in their own modeling files. This option should only be set to True for repositories you trust and in which you have read the code, as it will execute code present on the Hub on your local machine.
  • code_revision (str, optional, defaults to "main") — The specific revision to use for the code on the Hub, if the code leaves in a different repository than the rest of the model. It can be a branch name, a tag name, or a commit id, since we use a git-based system for storing models and other artifacts on huggingface.co, so revision can be any identifier allowed by git.
  • kwargs (additional keyword arguments, optional) — Can be used to update the configuration object (after it being loaded) and initiate the model (e.g., output_attentions=True). Behaves differently depending on whether a config is provided or automatically loaded:

    • If a configuration is provided with config, **kwargs will be directly passed to the underlying model’s __init__ method (we assume all relevant updates to the configuration have already been done)
    • If a configuration is not provided, kwargs will be first passed to the configuration class initialization function (from_pretrained()). Each key of kwargs that corresponds to a configuration attribute will be used to override said attribute with the supplied kwargs value. Remaining keys that do not correspond to any configuration attribute will be passed to the underlying model’s __init__ function.

Instantiate one of the model classes of the library (with a connectionist temporal classification head) from a pretrained model.

The model class to instantiate is selected based on the model_type property of the config object (either passed as an argument or loaded from pretrained_model_name_or_path if possible), or when it’s missing, by falling back to using pattern matching on pretrained_model_name_or_path:

  • data2vec-audio — Data2VecAudioForCTC (Data2VecAudio model)
  • hubert — HubertForCTC (Hubert model)
  • mctct — MCTCTForCTC (M-CTC-T model)
  • sew — SEWForCTC (SEW model)
  • sew-d — SEWDForCTC (SEW-D model)
  • unispeech — UniSpeechForCTC (UniSpeech model)
  • unispeech-sat — UniSpeechSatForCTC (UniSpeechSat model)
  • wav2vec2 — Wav2Vec2ForCTC (Wav2Vec2 model)
  • wav2vec2-bert — Wav2Vec2BertForCTC (Wav2Vec2-BERT model)
  • wav2vec2-conformer — Wav2Vec2ConformerForCTC (Wav2Vec2-Conformer model)
  • wavlm — WavLMForCTC (WavLM model)

The model is set in evaluation mode by default using model.eval() (so for instance, dropout modules are deactivated). To train the model, you should first set it back in training mode with model.train()

Examples:

>>> from transformers import AutoConfig, AutoModelForCTC

>>> # Download model and configuration from huggingface.co and cache.
>>> model = AutoModelForCTC.from_pretrained("google-bert/bert-base-cased")

>>> # Update configuration during loading
>>> model = AutoModelForCTC.from_pretrained("google-bert/bert-base-cased", output_attentions=True)
>>> model.config.output_attentions
True

>>> # Loading from a TF checkpoint file instead of a PyTorch model (slower)
>>> config = AutoConfig.from_pretrained("./tf_model/bert_tf_model_config.json")
>>> model = AutoModelForCTC.from_pretrained(
...     "./tf_model/bert_tf_checkpoint.ckpt.index", from_tf=True, config=config
... )

AutoModelForSpeechSeq2Seq

class transformers.AutoModelForSpeechSeq2Seq

< >

( *args **kwargs )

This is a generic model class that will be instantiated as one of the model classes of the library (with a sequence-to-sequence speech-to-text modeling head) when created with the from_pretrained() class method or the from_config() class method.

This class cannot be instantiated directly using __init__() (throws an error).

from_config

< >

( **kwargs )

Parameters

  • config (PretrainedConfig) — The model class to instantiate is selected based on the configuration class:

    • Pop2PianoConfig configuration class: Pop2PianoForConditionalGeneration (Pop2Piano model)
    • SeamlessM4TConfig configuration class: SeamlessM4TForSpeechToText (SeamlessM4T model)
    • SeamlessM4Tv2Config configuration class: SeamlessM4Tv2ForSpeechToText (SeamlessM4Tv2 model)
    • Speech2TextConfig configuration class: Speech2TextForConditionalGeneration (Speech2Text model)
    • SpeechEncoderDecoderConfig configuration class: SpeechEncoderDecoderModel (Speech Encoder decoder model)
    • SpeechT5Config configuration class: SpeechT5ForSpeechToText (SpeechT5 model)
    • WhisperConfig configuration class: WhisperForConditionalGeneration (Whisper model)
  • attn_implementation (str, optional) — The attention implementation to use in the model (if relevant). Can be any of "eager" (manual implementation of the attention), "sdpa" (using F.scaled_dot_product_attention), or "flash_attention_2" (using Dao-AILab/flash-attention). By default, if available, SDPA will be used for torch>=2.1.1. The default is otherwise the manual "eager" implementation.

Instantiates one of the model classes of the library (with a sequence-to-sequence speech-to-text modeling head) from a configuration.

Note: Loading a model from its configuration file does not load the model weights. It only affects the model’s configuration. Use from_pretrained() to load the model weights.

Examples:

>>> from transformers import AutoConfig, AutoModelForSpeechSeq2Seq

>>> # Download configuration from huggingface.co and cache.
>>> config = AutoConfig.from_pretrained("google-bert/bert-base-cased")
>>> model = AutoModelForSpeechSeq2Seq.from_config(config)

from_pretrained

< >

( *model_args **kwargs )

Parameters

  • pretrained_model_name_or_path (str or os.PathLike) — Can be either:

    • A string, the model id of a pretrained model hosted inside a model repo on huggingface.co.
    • A path to a directory containing model weights saved using save_pretrained(), e.g., ./my_model_directory/.
    • A path or url to a tensorflow index checkpoint file (e.g, ./tf_model/model.ckpt.index). In this case, from_tf should be set to True and a configuration object should be provided as config argument. This loading path is slower than converting the TensorFlow checkpoint in a PyTorch model using the provided conversion scripts and loading the PyTorch model afterwards.
  • model_args (additional positional arguments, optional) — Will be passed along to the underlying model __init__() method.
  • config (PretrainedConfig, optional) — Configuration for the model to use instead of an automatically loaded configuration. Configuration can be automatically loaded when:

    • The model is a model provided by the library (loaded with the model id string of a pretrained model).
    • The model was saved using save_pretrained() and is reloaded by supplying the save directory.
    • The model is loaded by supplying a local directory as pretrained_model_name_or_path and a configuration JSON file named config.json is found in the directory.
  • state_dict (Dict[str, torch.Tensor], optional) — A state dictionary to use instead of a state dictionary loaded from saved weights file.

    This option can be used if you want to create a model from a pretrained configuration but load your own weights. In this case though, you should check if using save_pretrained() and from_pretrained() is not a simpler option.

  • cache_dir (str or os.PathLike, optional) — Path to a directory in which a downloaded pretrained model configuration should be cached if the standard cache should not be used.
  • from_tf (bool, optional, defaults to False) — Load the model weights from a TensorFlow checkpoint save file (see docstring of pretrained_model_name_or_path argument).
  • force_download (bool, optional, defaults to False) — Whether or not to force the (re-)download of the model weights and configuration files, overriding the cached versions if they exist.
  • resume_download (bool, optional, defaults to False) — Whether or not to delete incompletely received files. Will attempt to resume the download if such a file exists.
  • proxies (Dict[str, str], optional) — A dictionary of proxy servers to use by protocol or endpoint, e.g., {'http': 'foo.bar:3128', 'http://hostname': 'foo.bar:4012'}. The proxies are used on each request.
  • output_loading_info(bool, optional, defaults to False) — Whether ot not to also return a dictionary containing missing keys, unexpected keys and error messages.
  • local_files_only(bool, optional, defaults to False) — Whether or not to only look at local files (e.g., not try downloading the model).
  • revision (str, optional, defaults to "main") — The specific model version to use. It can be a branch name, a tag name, or a commit id, since we use a git-based system for storing models and other artifacts on huggingface.co, so revision can be any identifier allowed by git.
  • trust_remote_code (bool, optional, defaults to False) — Whether or not to allow for custom models defined on the Hub in their own modeling files. This option should only be set to True for repositories you trust and in which you have read the code, as it will execute code present on the Hub on your local machine.
  • code_revision (str, optional, defaults to "main") — The specific revision to use for the code on the Hub, if the code leaves in a different repository than the rest of the model. It can be a branch name, a tag name, or a commit id, since we use a git-based system for storing models and other artifacts on huggingface.co, so revision can be any identifier allowed by git.
  • kwargs (additional keyword arguments, optional) — Can be used to update the configuration object (after it being loaded) and initiate the model (e.g., output_attentions=True). Behaves differently depending on whether a config is provided or automatically loaded:

    • If a configuration is provided with config, **kwargs will be directly passed to the underlying model’s __init__ method (we assume all relevant updates to the configuration have already been done)
    • If a configuration is not provided, kwargs will be first passed to the configuration class initialization function (from_pretrained()). Each key of kwargs that corresponds to a configuration attribute will be used to override said attribute with the supplied kwargs value. Remaining keys that do not correspond to any configuration attribute will be passed to the underlying model’s __init__ function.

Instantiate one of the model classes of the library (with a sequence-to-sequence speech-to-text modeling head) from a pretrained model.

The model class to instantiate is selected based on the model_type property of the config object (either passed as an argument or loaded from pretrained_model_name_or_path if possible), or when it’s missing, by falling back to using pattern matching on pretrained_model_name_or_path:

  • pop2piano — Pop2PianoForConditionalGeneration (Pop2Piano model)
  • seamless_m4t — SeamlessM4TForSpeechToText (SeamlessM4T model)
  • seamless_m4t_v2 — SeamlessM4Tv2ForSpeechToText (SeamlessM4Tv2 model)
  • speech-encoder-decoder — SpeechEncoderDecoderModel (Speech Encoder decoder model)
  • speech_to_text — Speech2TextForConditionalGeneration (Speech2Text model)
  • speecht5 — SpeechT5ForSpeechToText (SpeechT5 model)
  • whisper — WhisperForConditionalGeneration (Whisper model)

The model is set in evaluation mode by default using model.eval() (so for instance, dropout modules are deactivated). To train the model, you should first set it back in training mode with model.train()

Examples:

>>> from transformers import AutoConfig, AutoModelForSpeechSeq2Seq

>>> # Download model and configuration from huggingface.co and cache.
>>> model = AutoModelForSpeechSeq2Seq.from_pretrained("google-bert/bert-base-cased")

>>> # Update configuration during loading
>>> model = AutoModelForSpeechSeq2Seq.from_pretrained("google-bert/bert-base-cased", output_attentions=True)
>>> model.config.output_attentions
True

>>> # Loading from a TF checkpoint file instead of a PyTorch model (slower)
>>> config = AutoConfig.from_pretrained("./tf_model/bert_tf_model_config.json")
>>> model = AutoModelForSpeechSeq2Seq.from_pretrained(
...     "./tf_model/bert_tf_checkpoint.ckpt.index", from_tf=True, config=config
... )

TFAutoModelForSpeechSeq2Seq

class transformers.TFAutoModelForSpeechSeq2Seq

< >

( *args **kwargs )

This is a generic model class that will be instantiated as one of the model classes of the library (with a sequence-to-sequence speech-to-text modeling head) when created with the from_pretrained() class method or the from_config() class method.

This class cannot be instantiated directly using __init__() (throws an error).

from_config

< >

( **kwargs )

Parameters

  • config (PretrainedConfig) — The model class to instantiate is selected based on the configuration class:

    • Speech2TextConfig configuration class: TFSpeech2TextForConditionalGeneration (Speech2Text model)
    • WhisperConfig configuration class: TFWhisperForConditionalGeneration (Whisper model)
  • attn_implementation (str, optional) — The attention implementation to use in the model (if relevant). Can be any of "eager" (manual implementation of the attention), "sdpa" (using F.scaled_dot_product_attention), or "flash_attention_2" (using Dao-AILab/flash-attention). By default, if available, SDPA will be used for torch>=2.1.1. The default is otherwise the manual "eager" implementation.

Instantiates one of the model classes of the library (with a sequence-to-sequence speech-to-text modeling head) from a configuration.

Note: Loading a model from its configuration file does not load the model weights. It only affects the model’s configuration. Use from_pretrained() to load the model weights.

Examples:

>>> from transformers import AutoConfig, TFAutoModelForSpeechSeq2Seq

>>> # Download configuration from huggingface.co and cache.
>>> config = AutoConfig.from_pretrained("google-bert/bert-base-cased")
>>> model = TFAutoModelForSpeechSeq2Seq.from_config(config)

from_pretrained

< >

( *model_args **kwargs )

Parameters

  • pretrained_model_name_or_path (str or os.PathLike) — Can be either:

    • A string, the model id of a pretrained model hosted inside a model repo on huggingface.co.
    • A path to a directory containing model weights saved using save_pretrained(), e.g., ./my_model_directory/.
    • A path or url to a PyTorch state_dict save file (e.g, ./pt_model/pytorch_model.bin). In this case, from_pt should be set to True and a configuration object should be provided as config argument. This loading path is slower than converting the PyTorch model in a TensorFlow model using the provided conversion scripts and loading the TensorFlow model afterwards.
  • model_args (additional positional arguments, optional) — Will be passed along to the underlying model __init__() method.
  • config (PretrainedConfig, optional) — Configuration for the model to use instead of an automatically loaded configuration. Configuration can be automatically loaded when:

    • The model is a model provided by the library (loaded with the model id string of a pretrained model).
    • The model was saved using save_pretrained() and is reloaded by supplying the save directory.
    • The model is loaded by supplying a local directory as pretrained_model_name_or_path and a configuration JSON file named config.json is found in the directory.
  • cache_dir (str or os.PathLike, optional) — Path to a directory in which a downloaded pretrained model configuration should be cached if the standard cache should not be used.
  • from_pt (bool, optional, defaults to False) — Load the model weights from a PyTorch checkpoint save file (see docstring of pretrained_model_name_or_path argument).
  • force_download (bool, optional, defaults to False) — Whether or not to force the (re-)download of the model weights and configuration files, overriding the cached versions if they exist.
  • resume_download (bool, optional, defaults to False) — Whether or not to delete incompletely received files. Will attempt to resume the download if such a file exists.
  • proxies (Dict[str, str], optional) — A dictionary of proxy servers to use by protocol or endpoint, e.g., {'http': 'foo.bar:3128', 'http://hostname': 'foo.bar:4012'}. The proxies are used on each request.
  • output_loading_info(bool, optional, defaults to False) — Whether ot not to also return a dictionary containing missing keys, unexpected keys and error messages.
  • local_files_only(bool, optional, defaults to False) — Whether or not to only look at local files (e.g., not try downloading the model).
  • revision (str, optional, defaults to "main") — The specific model version to use. It can be a branch name, a tag name, or a commit id, since we use a git-based system for storing models and other artifacts on huggingface.co, so revision can be any identifier allowed by git.
  • trust_remote_code (bool, optional, defaults to False) — Whether or not to allow for custom models defined on the Hub in their own modeling files. This option should only be set to True for repositories you trust and in which you have read the code, as it will execute code present on the Hub on your local machine.
  • code_revision (str, optional, defaults to "main") — The specific revision to use for the code on the Hub, if the code leaves in a different repository than the rest of the model. It can be a branch name, a tag name, or a commit id, since we use a git-based system for storing models and other artifacts on huggingface.co, so revision can be any identifier allowed by git.
  • kwargs (additional keyword arguments, optional) — Can be used to update the configuration object (after it being loaded) and initiate the model (e.g., output_attentions=True). Behaves differently depending on whether a config is provided or automatically loaded:

    • If a configuration is provided with config, **kwargs will be directly passed to the underlying model’s __init__ method (we assume all relevant updates to the configuration have already been done)
    • If a configuration is not provided, kwargs will be first passed to the configuration class initialization function (from_pretrained()). Each key of kwargs that corresponds to a configuration attribute will be used to override said attribute with the supplied kwargs value. Remaining keys that do not correspond to any configuration attribute will be passed to the underlying model’s __init__ function.

Instantiate one of the model classes of the library (with a sequence-to-sequence speech-to-text modeling head) from a pretrained model.

The model class to instantiate is selected based on the model_type property of the config object (either passed as an argument or loaded from pretrained_model_name_or_path if possible), or when it’s missing, by falling back to using pattern matching on pretrained_model_name_or_path:

  • speech_to_text — TFSpeech2TextForConditionalGeneration (Speech2Text model)
  • whisper — TFWhisperForConditionalGeneration (Whisper model)

Examples:

>>> from transformers import AutoConfig, TFAutoModelForSpeechSeq2Seq

>>> # Download model and configuration from huggingface.co and cache.
>>> model = TFAutoModelForSpeechSeq2Seq.from_pretrained("google-bert/bert-base-cased")

>>> # Update configuration during loading
>>> model = TFAutoModelForSpeechSeq2Seq.from_pretrained("google-bert/bert-base-cased", output_attentions=True)
>>> model.config.output_attentions
True

>>> # Loading from a PyTorch checkpoint file instead of a TensorFlow model (slower)
>>> config = AutoConfig.from_pretrained("./pt_model/bert_pt_model_config.json")
>>> model = TFAutoModelForSpeechSeq2Seq.from_pretrained(
...     "./pt_model/bert_pytorch_model.bin", from_pt=True, config=config
... )

FlaxAutoModelForSpeechSeq2Seq

class transformers.FlaxAutoModelForSpeechSeq2Seq

< >

( *args **kwargs )

This is a generic model class that will be instantiated as one of the model classes of the library (with a sequence-to-sequence speech-to-text modeling head) when created with the from_pretrained() class method or the from_config() class method.

This class cannot be instantiated directly using __init__() (throws an error).

from_config

< >

( **kwargs )

Parameters

  • config (PretrainedConfig) — The model class to instantiate is selected based on the configuration class:

    • SpeechEncoderDecoderConfig configuration class: FlaxSpeechEncoderDecoderModel (Speech Encoder decoder model)
    • WhisperConfig configuration class: FlaxWhisperForConditionalGeneration (Whisper model)
  • attn_implementation (str, optional) — The attention implementation to use in the model (if relevant). Can be any of "eager" (manual implementation of the attention), "sdpa" (using F.scaled_dot_product_attention), or "flash_attention_2" (using Dao-AILab/flash-attention). By default, if available, SDPA will be used for torch>=2.1.1. The default is otherwise the manual "eager" implementation.

Instantiates one of the model classes of the library (with a sequence-to-sequence speech-to-text modeling head) from a configuration.

Note: Loading a model from its configuration file does not load the model weights. It only affects the model’s configuration. Use from_pretrained() to load the model weights.

Examples:

>>> from transformers import AutoConfig, FlaxAutoModelForSpeechSeq2Seq

>>> # Download configuration from huggingface.co and cache.
>>> config = AutoConfig.from_pretrained("google-bert/bert-base-cased")
>>> model = FlaxAutoModelForSpeechSeq2Seq.from_config(config)

from_pretrained

< >

( *model_args **kwargs )

Parameters

  • pretrained_model_name_or_path (str or os.PathLike) — Can be either:

    • A string, the model id of a pretrained model hosted inside a model repo on huggingface.co.
    • A path to a directory containing model weights saved using save_pretrained(), e.g., ./my_model_directory/.
    • A path or url to a PyTorch state_dict save file (e.g, ./pt_model/pytorch_model.bin). In this case, from_pt should be set to True and a configuration object should be provided as config argument. This loading path is slower than converting the PyTorch model in a TensorFlow model using the provided conversion scripts and loading the TensorFlow model afterwards.
  • model_args (additional positional arguments, optional) — Will be passed along to the underlying model __init__() method.
  • config (PretrainedConfig, optional) — Configuration for the model to use instead of an automatically loaded configuration. Configuration can be automatically loaded when:

    • The model is a model provided by the library (loaded with the model id string of a pretrained model).
    • The model was saved using save_pretrained() and is reloaded by supplying the save directory.
    • The model is loaded by supplying a local directory as pretrained_model_name_or_path and a configuration JSON file named config.json is found in the directory.
  • cache_dir (str or os.PathLike, optional) — Path to a directory in which a downloaded pretrained model configuration should be cached if the standard cache should not be used.
  • from_pt (bool, optional, defaults to False) — Load the model weights from a PyTorch checkpoint save file (see docstring of pretrained_model_name_or_path argument).
  • force_download (bool, optional, defaults to False) — Whether or not to force the (re-)download of the model weights and configuration files, overriding the cached versions if they exist.
  • resume_download (bool, optional, defaults to False) — Whether or not to delete incompletely received files. Will attempt to resume the download if such a file exists.
  • proxies (Dict[str, str], optional) — A dictionary of proxy servers to use by protocol or endpoint, e.g., {'http': 'foo.bar:3128', 'http://hostname': 'foo.bar:4012'}. The proxies are used on each request.
  • output_loading_info(bool, optional, defaults to False) — Whether ot not to also return a dictionary containing missing keys, unexpected keys and error messages.
  • local_files_only(bool, optional, defaults to False) — Whether or not to only look at local files (e.g., not try downloading the model).
  • revision (str, optional, defaults to "main") — The specific model version to use. It can be a branch name, a tag name, or a commit id, since we use a git-based system for storing models and other artifacts on huggingface.co, so revision can be any identifier allowed by git.
  • trust_remote_code (bool, optional, defaults to False) — Whether or not to allow for custom models defined on the Hub in their own modeling files. This option should only be set to True for repositories you trust and in which you have read the code, as it will execute code present on the Hub on your local machine.
  • code_revision (str, optional, defaults to "main") — The specific revision to use for the code on the Hub, if the code leaves in a different repository than the rest of the model. It can be a branch name, a tag name, or a commit id, since we use a git-based system for storing models and other artifacts on huggingface.co, so revision can be any identifier allowed by git.
  • kwargs (additional keyword arguments, optional) — Can be used to update the configuration object (after it being loaded) and initiate the model (e.g., output_attentions=True). Behaves differently depending on whether a config is provided or automatically loaded:

    • If a configuration is provided with config, **kwargs will be directly passed to the underlying model’s __init__ method (we assume all relevant updates to the configuration have already been done)
    • If a configuration is not provided, kwargs will be first passed to the configuration class initialization function (from_pretrained()). Each key of kwargs that corresponds to a configuration attribute will be used to override said attribute with the supplied kwargs value. Remaining keys that do not correspond to any configuration attribute will be passed to the underlying model’s __init__ function.

Instantiate one of the model classes of the library (with a sequence-to-sequence speech-to-text modeling head) from a pretrained model.

The model class to instantiate is selected based on the model_type property of the config object (either passed as an argument or loaded from pretrained_model_name_or_path if possible), or when it’s missing, by falling back to using pattern matching on pretrained_model_name_or_path:

  • speech-encoder-decoder — FlaxSpeechEncoderDecoderModel (Speech Encoder decoder model)
  • whisper — FlaxWhisperForConditionalGeneration (Whisper model)

Examples:

>>> from transformers import AutoConfig, FlaxAutoModelForSpeechSeq2Seq

>>> # Download model and configuration from huggingface.co and cache.
>>> model = FlaxAutoModelForSpeechSeq2Seq.from_pretrained("google-bert/bert-base-cased")

>>> # Update configuration during loading
>>> model = FlaxAutoModelForSpeechSeq2Seq.from_pretrained("google-bert/bert-base-cased", output_attentions=True)
>>> model.config.output_attentions
True

>>> # Loading from a PyTorch checkpoint file instead of a TensorFlow model (slower)
>>> config = AutoConfig.from_pretrained("./pt_model/bert_pt_model_config.json")
>>> model = FlaxAutoModelForSpeechSeq2Seq.from_pretrained(
...     "./pt_model/bert_pytorch_model.bin", from_pt=True, config=config
... )

AutoModelForAudioXVector

class transformers.AutoModelForAudioXVector

< >

( *args **kwargs )

This is a generic model class that will be instantiated as one of the model classes of the library (with a audio retrieval via x-vector head) when created with the from_pretrained() class method or the from_config() class method.

This class cannot be instantiated directly using __init__() (throws an error).

from_config

< >

( **kwargs )

Parameters

  • config (PretrainedConfig) — The model class to instantiate is selected based on the configuration class:

    • Data2VecAudioConfig configuration class: Data2VecAudioForXVector (Data2VecAudio model)
    • UniSpeechSatConfig configuration class: UniSpeechSatForXVector (UniSpeechSat model)
    • Wav2Vec2BertConfig configuration class: Wav2Vec2BertForXVector (Wav2Vec2-BERT model)
    • Wav2Vec2Config configuration class: Wav2Vec2ForXVector (Wav2Vec2 model)
    • Wav2Vec2ConformerConfig configuration class: Wav2Vec2ConformerForXVector (Wav2Vec2-Conformer model)
    • WavLMConfig configuration class: WavLMForXVector (WavLM model)
  • attn_implementation (str, optional) — The attention implementation to use in the model (if relevant). Can be any of "eager" (manual implementation of the attention), "sdpa" (using F.scaled_dot_product_attention), or "flash_attention_2" (using Dao-AILab/flash-attention). By default, if available, SDPA will be used for torch>=2.1.1. The default is otherwise the manual "eager" implementation.

Instantiates one of the model classes of the library (with a audio retrieval via x-vector head) from a configuration.

Note: Loading a model from its configuration file does not load the model weights. It only affects the model’s configuration. Use from_pretrained() to load the model weights.

Examples:

>>> from transformers import AutoConfig, AutoModelForAudioXVector

>>> # Download configuration from huggingface.co and cache.
>>> config = AutoConfig.from_pretrained("google-bert/bert-base-cased")
>>> model = AutoModelForAudioXVector.from_config(config)

from_pretrained

< >

( *model_args **kwargs )

Parameters

  • pretrained_model_name_or_path (str or os.PathLike) — Can be either:

    • A string, the model id of a pretrained model hosted inside a model repo on huggingface.co.
    • A path to a directory containing model weights saved using save_pretrained(), e.g., ./my_model_directory/.
    • A path or url to a tensorflow index checkpoint file (e.g, ./tf_model/model.ckpt.index). In this case, from_tf should be set to True and a configuration object should be provided as config argument. This loading path is slower than converting the TensorFlow checkpoint in a PyTorch model using the provided conversion scripts and loading the PyTorch model afterwards.
  • model_args (additional positional arguments, optional) — Will be passed along to the underlying model __init__() method.
  • config (PretrainedConfig, optional) — Configuration for the model to use instead of an automatically loaded configuration. Configuration can be automatically loaded when:

    • The model is a model provided by the library (loaded with the model id string of a pretrained model).
    • The model was saved using save_pretrained() and is reloaded by supplying the save directory.
    • The model is loaded by supplying a local directory as pretrained_model_name_or_path and a configuration JSON file named config.json is found in the directory.
  • state_dict (Dict[str, torch.Tensor], optional) — A state dictionary to use instead of a state dictionary loaded from saved weights file.

    This option can be used if you want to create a model from a pretrained configuration but load your own weights. In this case though, you should check if using save_pretrained() and from_pretrained() is not a simpler option.

  • cache_dir (str or os.PathLike, optional) — Path to a directory in which a downloaded pretrained model configuration should be cached if the standard cache should not be used.
  • from_tf (bool, optional, defaults to False) — Load the model weights from a TensorFlow checkpoint save file (see docstring of pretrained_model_name_or_path argument).
  • force_download (bool, optional, defaults to False) — Whether or not to force the (re-)download of the model weights and configuration files, overriding the cached versions if they exist.
  • resume_download (bool, optional, defaults to False) — Whether or not to delete incompletely received files. Will attempt to resume the download if such a file exists.
  • proxies (Dict[str, str], optional) — A dictionary of proxy servers to use by protocol or endpoint, e.g., {'http': 'foo.bar:3128', 'http://hostname': 'foo.bar:4012'}. The proxies are used on each request.
  • output_loading_info(bool, optional, defaults to False) — Whether ot not to also return a dictionary containing missing keys, unexpected keys and error messages.
  • local_files_only(bool, optional, defaults to False) — Whether or not to only look at local files (e.g., not try downloading the model).
  • revision (str, optional, defaults to "main") — The specific model version to use. It can be a branch name, a tag name, or a commit id, since we use a git-based system for storing models and other artifacts on huggingface.co, so revision can be any identifier allowed by git.
  • trust_remote_code (bool, optional, defaults to False) — Whether or not to allow for custom models defined on the Hub in their own modeling files. This option should only be set to True for repositories you trust and in which you have read the code, as it will execute code present on the Hub on your local machine.
  • code_revision (str, optional, defaults to "main") — The specific revision to use for the code on the Hub, if the code leaves in a different repository than the rest of the model. It can be a branch name, a tag name, or a commit id, since we use a git-based system for storing models and other artifacts on huggingface.co, so revision can be any identifier allowed by git.
  • kwargs (additional keyword arguments, optional) — Can be used to update the configuration object (after it being loaded) and initiate the model (e.g., output_attentions=True). Behaves differently depending on whether a config is provided or automatically loaded:

    • If a configuration is provided with config, **kwargs will be directly passed to the underlying model’s __init__ method (we assume all relevant updates to the configuration have already been done)
    • If a configuration is not provided, kwargs will be first passed to the configuration class initialization function (from_pretrained()). Each key of kwargs that corresponds to a configuration attribute will be used to override said attribute with the supplied kwargs value. Remaining keys that do not correspond to any configuration attribute will be passed to the underlying model’s __init__ function.

Instantiate one of the model classes of the library (with a audio retrieval via x-vector head) from a pretrained model.

The model class to instantiate is selected based on the model_type property of the config object (either passed as an argument or loaded from pretrained_model_name_or_path if possible), or when it’s missing, by falling back to using pattern matching on pretrained_model_name_or_path:

  • data2vec-audio — Data2VecAudioForXVector (Data2VecAudio model)
  • unispeech-sat — UniSpeechSatForXVector (UniSpeechSat model)
  • wav2vec2 — Wav2Vec2ForXVector (Wav2Vec2 model)
  • wav2vec2-bert — Wav2Vec2BertForXVector (Wav2Vec2-BERT model)
  • wav2vec2-conformer — Wav2Vec2ConformerForXVector (Wav2Vec2-Conformer model)
  • wavlm — WavLMForXVector (WavLM model)

The model is set in evaluation mode by default using model.eval() (so for instance, dropout modules are deactivated). To train the model, you should first set it back in training mode with model.train()

Examples:

>>> from transformers import AutoConfig, AutoModelForAudioXVector

>>> # Download model and configuration from huggingface.co and cache.
>>> model = AutoModelForAudioXVector.from_pretrained("google-bert/bert-base-cased")

>>> # Update configuration during loading
>>> model = AutoModelForAudioXVector.from_pretrained("google-bert/bert-base-cased", output_attentions=True)
>>> model.config.output_attentions
True

>>> # Loading from a TF checkpoint file instead of a PyTorch model (slower)
>>> config = AutoConfig.from_pretrained("./tf_model/bert_tf_model_config.json")
>>> model = AutoModelForAudioXVector.from_pretrained(
...     "./tf_model/bert_tf_checkpoint.ckpt.index", from_tf=True, config=config
... )

AutoModelForTextToSpectrogram

class transformers.AutoModelForTextToSpectrogram

< >

( *args **kwargs )

AutoModelForTextToWaveform

class transformers.AutoModelForTextToWaveform

< >

( *args **kwargs )

Multimodal

以下の自動クラスは、次のマルチモーダルタスクに利用可能です。

AutoModelForTableQuestionAnswering

class transformers.AutoModelForTableQuestionAnswering

< >

( *args **kwargs )

This is a generic model class that will be instantiated as one of the model classes of the library (with a table question answering head) when created with the from_pretrained() class method or the from_config() class method.

This class cannot be instantiated directly using __init__() (throws an error).

from_config

< >

( **kwargs )

Parameters

  • config (PretrainedConfig) — The model class to instantiate is selected based on the configuration class:

    • TapasConfig configuration class: TapasForQuestionAnswering (TAPAS model)
  • attn_implementation (str, optional) — The attention implementation to use in the model (if relevant). Can be any of "eager" (manual implementation of the attention), "sdpa" (using F.scaled_dot_product_attention), or "flash_attention_2" (using Dao-AILab/flash-attention). By default, if available, SDPA will be used for torch>=2.1.1. The default is otherwise the manual "eager" implementation.

Instantiates one of the model classes of the library (with a table question answering head) from a configuration.

Note: Loading a model from its configuration file does not load the model weights. It only affects the model’s configuration. Use from_pretrained() to load the model weights.

Examples:

>>> from transformers import AutoConfig, AutoModelForTableQuestionAnswering

>>> # Download configuration from huggingface.co and cache.
>>> config = AutoConfig.from_pretrained("google/tapas-base-finetuned-wtq")
>>> model = AutoModelForTableQuestionAnswering.from_config(config)

from_pretrained

< >

( *model_args **kwargs )

Parameters

  • pretrained_model_name_or_path (str or os.PathLike) — Can be either:

    • A string, the model id of a pretrained model hosted inside a model repo on huggingface.co.
    • A path to a directory containing model weights saved using save_pretrained(), e.g., ./my_model_directory/.
    • A path or url to a tensorflow index checkpoint file (e.g, ./tf_model/model.ckpt.index). In this case, from_tf should be set to True and a configuration object should be provided as config argument. This loading path is slower than converting the TensorFlow checkpoint in a PyTorch model using the provided conversion scripts and loading the PyTorch model afterwards.
  • model_args (additional positional arguments, optional) — Will be passed along to the underlying model __init__() method.
  • config (PretrainedConfig, optional) — Configuration for the model to use instead of an automatically loaded configuration. Configuration can be automatically loaded when:

    • The model is a model provided by the library (loaded with the model id string of a pretrained model).
    • The model was saved using save_pretrained() and is reloaded by supplying the save directory.
    • The model is loaded by supplying a local directory as pretrained_model_name_or_path and a configuration JSON file named config.json is found in the directory.
  • state_dict (Dict[str, torch.Tensor], optional) — A state dictionary to use instead of a state dictionary loaded from saved weights file.

    This option can be used if you want to create a model from a pretrained configuration but load your own weights. In this case though, you should check if using save_pretrained() and from_pretrained() is not a simpler option.

  • cache_dir (str or os.PathLike, optional) — Path to a directory in which a downloaded pretrained model configuration should be cached if the standard cache should not be used.
  • from_tf (bool, optional, defaults to False) — Load the model weights from a TensorFlow checkpoint save file (see docstring of pretrained_model_name_or_path argument).
  • force_download (bool, optional, defaults to False) — Whether or not to force the (re-)download of the model weights and configuration files, overriding the cached versions if they exist.
  • resume_download (bool, optional, defaults to False) — Whether or not to delete incompletely received files. Will attempt to resume the download if such a file exists.
  • proxies (Dict[str, str], optional) — A dictionary of proxy servers to use by protocol or endpoint, e.g., {'http': 'foo.bar:3128', 'http://hostname': 'foo.bar:4012'}. The proxies are used on each request.
  • output_loading_info(bool, optional, defaults to False) — Whether ot not to also return a dictionary containing missing keys, unexpected keys and error messages.
  • local_files_only(bool, optional, defaults to False) — Whether or not to only look at local files (e.g., not try downloading the model).
  • revision (str, optional, defaults to "main") — The specific model version to use. It can be a branch name, a tag name, or a commit id, since we use a git-based system for storing models and other artifacts on huggingface.co, so revision can be any identifier allowed by git.
  • trust_remote_code (bool, optional, defaults to False) — Whether or not to allow for custom models defined on the Hub in their own modeling files. This option should only be set to True for repositories you trust and in which you have read the code, as it will execute code present on the Hub on your local machine.
  • code_revision (str, optional, defaults to "main") — The specific revision to use for the code on the Hub, if the code leaves in a different repository than the rest of the model. It can be a branch name, a tag name, or a commit id, since we use a git-based system for storing models and other artifacts on huggingface.co, so revision can be any identifier allowed by git.
  • kwargs (additional keyword arguments, optional) — Can be used to update the configuration object (after it being loaded) and initiate the model (e.g., output_attentions=True). Behaves differently depending on whether a config is provided or automatically loaded:

    • If a configuration is provided with config, **kwargs will be directly passed to the underlying model’s __init__ method (we assume all relevant updates to the configuration have already been done)
    • If a configuration is not provided, kwargs will be first passed to the configuration class initialization function (from_pretrained()). Each key of kwargs that corresponds to a configuration attribute will be used to override said attribute with the supplied kwargs value. Remaining keys that do not correspond to any configuration attribute will be passed to the underlying model’s __init__ function.

Instantiate one of the model classes of the library (with a table question answering head) from a pretrained model.

The model class to instantiate is selected based on the model_type property of the config object (either passed as an argument or loaded from pretrained_model_name_or_path if possible), or when it’s missing, by falling back to using pattern matching on pretrained_model_name_or_path:

  • tapas — TapasForQuestionAnswering (TAPAS model)

The model is set in evaluation mode by default using model.eval() (so for instance, dropout modules are deactivated). To train the model, you should first set it back in training mode with model.train()

Examples:

>>> from transformers import AutoConfig, AutoModelForTableQuestionAnswering

>>> # Download model and configuration from huggingface.co and cache.
>>> model = AutoModelForTableQuestionAnswering.from_pretrained("google/tapas-base-finetuned-wtq")

>>> # Update configuration during loading
>>> model = AutoModelForTableQuestionAnswering.from_pretrained("google/tapas-base-finetuned-wtq", output_attentions=True)
>>> model.config.output_attentions
True

>>> # Loading from a TF checkpoint file instead of a PyTorch model (slower)
>>> config = AutoConfig.from_pretrained("./tf_model/tapas_tf_model_config.json")
>>> model = AutoModelForTableQuestionAnswering.from_pretrained(
...     "./tf_model/tapas_tf_checkpoint.ckpt.index", from_tf=True, config=config
... )

TFAutoModelForTableQuestionAnswering

class transformers.TFAutoModelForTableQuestionAnswering

< >

( *args **kwargs )

This is a generic model class that will be instantiated as one of the model classes of the library (with a table question answering head) when created with the from_pretrained() class method or the from_config() class method.

This class cannot be instantiated directly using __init__() (throws an error).

from_config

< >

( **kwargs )

Parameters

  • config (PretrainedConfig) — The model class to instantiate is selected based on the configuration class:

    • TapasConfig configuration class: TFTapasForQuestionAnswering (TAPAS model)
  • attn_implementation (str, optional) — The attention implementation to use in the model (if relevant). Can be any of "eager" (manual implementation of the attention), "sdpa" (using F.scaled_dot_product_attention), or "flash_attention_2" (using Dao-AILab/flash-attention). By default, if available, SDPA will be used for torch>=2.1.1. The default is otherwise the manual "eager" implementation.

Instantiates one of the model classes of the library (with a table question answering head) from a configuration.

Note: Loading a model from its configuration file does not load the model weights. It only affects the model’s configuration. Use from_pretrained() to load the model weights.

Examples:

>>> from transformers import AutoConfig, TFAutoModelForTableQuestionAnswering

>>> # Download configuration from huggingface.co and cache.
>>> config = AutoConfig.from_pretrained("google/tapas-base-finetuned-wtq")
>>> model = TFAutoModelForTableQuestionAnswering.from_config(config)

from_pretrained

< >

( *model_args **kwargs )

Parameters

  • pretrained_model_name_or_path (str or os.PathLike) — Can be either:

    • A string, the model id of a pretrained model hosted inside a model repo on huggingface.co.
    • A path to a directory containing model weights saved using save_pretrained(), e.g., ./my_model_directory/.
    • A path or url to a PyTorch state_dict save file (e.g, ./pt_model/pytorch_model.bin). In this case, from_pt should be set to True and a configuration object should be provided as config argument. This loading path is slower than converting the PyTorch model in a TensorFlow model using the provided conversion scripts and loading the TensorFlow model afterwards.
  • model_args (additional positional arguments, optional) — Will be passed along to the underlying model __init__() method.
  • config (PretrainedConfig, optional) — Configuration for the model to use instead of an automatically loaded configuration. Configuration can be automatically loaded when:

    • The model is a model provided by the library (loaded with the model id string of a pretrained model).
    • The model was saved using save_pretrained() and is reloaded by supplying the save directory.
    • The model is loaded by supplying a local directory as pretrained_model_name_or_path and a configuration JSON file named config.json is found in the directory.
  • cache_dir (str or os.PathLike, optional) — Path to a directory in which a downloaded pretrained model configuration should be cached if the standard cache should not be used.
  • from_pt (bool, optional, defaults to False) — Load the model weights from a PyTorch checkpoint save file (see docstring of pretrained_model_name_or_path argument).
  • force_download (bool, optional, defaults to False) — Whether or not to force the (re-)download of the model weights and configuration files, overriding the cached versions if they exist.
  • resume_download (bool, optional, defaults to False) — Whether or not to delete incompletely received files. Will attempt to resume the download if such a file exists.
  • proxies (Dict[str, str], optional) — A dictionary of proxy servers to use by protocol or endpoint, e.g., {'http': 'foo.bar:3128', 'http://hostname': 'foo.bar:4012'}. The proxies are used on each request.
  • output_loading_info(bool, optional, defaults to False) — Whether ot not to also return a dictionary containing missing keys, unexpected keys and error messages.
  • local_files_only(bool, optional, defaults to False) — Whether or not to only look at local files (e.g., not try downloading the model).
  • revision (str, optional, defaults to "main") — The specific model version to use. It can be a branch name, a tag name, or a commit id, since we use a git-based system for storing models and other artifacts on huggingface.co, so revision can be any identifier allowed by git.
  • trust_remote_code (bool, optional, defaults to False) — Whether or not to allow for custom models defined on the Hub in their own modeling files. This option should only be set to True for repositories you trust and in which you have read the code, as it will execute code present on the Hub on your local machine.
  • code_revision (str, optional, defaults to "main") — The specific revision to use for the code on the Hub, if the code leaves in a different repository than the rest of the model. It can be a branch name, a tag name, or a commit id, since we use a git-based system for storing models and other artifacts on huggingface.co, so revision can be any identifier allowed by git.
  • kwargs (additional keyword arguments, optional) — Can be used to update the configuration object (after it being loaded) and initiate the model (e.g., output_attentions=True). Behaves differently depending on whether a config is provided or automatically loaded:

    • If a configuration is provided with config, **kwargs will be directly passed to the underlying model’s __init__ method (we assume all relevant updates to the configuration have already been done)
    • If a configuration is not provided, kwargs will be first passed to the configuration class initialization function (from_pretrained()). Each key of kwargs that corresponds to a configuration attribute will be used to override said attribute with the supplied kwargs value. Remaining keys that do not correspond to any configuration attribute will be passed to the underlying model’s __init__ function.

Instantiate one of the model classes of the library (with a table question answering head) from a pretrained model.

The model class to instantiate is selected based on the model_type property of the config object (either passed as an argument or loaded from pretrained_model_name_or_path if possible), or when it’s missing, by falling back to using pattern matching on pretrained_model_name_or_path:

  • tapas — TFTapasForQuestionAnswering (TAPAS model)

Examples:

>>> from transformers import AutoConfig, TFAutoModelForTableQuestionAnswering

>>> # Download model and configuration from huggingface.co and cache.
>>> model = TFAutoModelForTableQuestionAnswering.from_pretrained("google/tapas-base-finetuned-wtq")

>>> # Update configuration during loading
>>> model = TFAutoModelForTableQuestionAnswering.from_pretrained("google/tapas-base-finetuned-wtq", output_attentions=True)
>>> model.config.output_attentions
True

>>> # Loading from a PyTorch checkpoint file instead of a TensorFlow model (slower)
>>> config = AutoConfig.from_pretrained("./pt_model/tapas_pt_model_config.json")
>>> model = TFAutoModelForTableQuestionAnswering.from_pretrained(
...     "./pt_model/tapas_pytorch_model.bin", from_pt=True, config=config
... )

AutoModelForDocumentQuestionAnswering

class transformers.AutoModelForDocumentQuestionAnswering

< >

( *args **kwargs )

This is a generic model class that will be instantiated as one of the model classes of the library (with a document question answering head) when created with the from_pretrained() class method or the from_config() class method.

This class cannot be instantiated directly using __init__() (throws an error).

from_config

< >

( **kwargs )

Parameters

  • config (PretrainedConfig) — The model class to instantiate is selected based on the configuration class:

    • LayoutLMConfig configuration class: LayoutLMForQuestionAnswering (LayoutLM model)
    • LayoutLMv2Config configuration class: LayoutLMv2ForQuestionAnswering (LayoutLMv2 model)
    • LayoutLMv3Config configuration class: LayoutLMv3ForQuestionAnswering (LayoutLMv3 model)
  • attn_implementation (str, optional) — The attention implementation to use in the model (if relevant). Can be any of "eager" (manual implementation of the attention), "sdpa" (using F.scaled_dot_product_attention), or "flash_attention_2" (using Dao-AILab/flash-attention). By default, if available, SDPA will be used for torch>=2.1.1. The default is otherwise the manual "eager" implementation.

Instantiates one of the model classes of the library (with a document question answering head) from a configuration.

Note: Loading a model from its configuration file does not load the model weights. It only affects the model’s configuration. Use from_pretrained() to load the model weights.

Examples:

>>> from transformers import AutoConfig, AutoModelForDocumentQuestionAnswering

>>> # Download configuration from huggingface.co and cache.
>>> config = AutoConfig.from_pretrained("impira/layoutlm-document-qa", revision="52e01b3")
>>> model = AutoModelForDocumentQuestionAnswering.from_config(config)

from_pretrained

< >

( *model_args **kwargs )

Parameters

  • pretrained_model_name_or_path (str or os.PathLike) — Can be either:

    • A string, the model id of a pretrained model hosted inside a model repo on huggingface.co.
    • A path to a directory containing model weights saved using save_pretrained(), e.g., ./my_model_directory/.
    • A path or url to a tensorflow index checkpoint file (e.g, ./tf_model/model.ckpt.index). In this case, from_tf should be set to True and a configuration object should be provided as config argument. This loading path is slower than converting the TensorFlow checkpoint in a PyTorch model using the provided conversion scripts and loading the PyTorch model afterwards.
  • model_args (additional positional arguments, optional) — Will be passed along to the underlying model __init__() method.
  • config (PretrainedConfig, optional) — Configuration for the model to use instead of an automatically loaded configuration. Configuration can be automatically loaded when:

    • The model is a model provided by the library (loaded with the model id string of a pretrained model).
    • The model was saved using save_pretrained() and is reloaded by supplying the save directory.
    • The model is loaded by supplying a local directory as pretrained_model_name_or_path and a configuration JSON file named config.json is found in the directory.
  • state_dict (Dict[str, torch.Tensor], optional) — A state dictionary to use instead of a state dictionary loaded from saved weights file.

    This option can be used if you want to create a model from a pretrained configuration but load your own weights. In this case though, you should check if using save_pretrained() and from_pretrained() is not a simpler option.

  • cache_dir (str or os.PathLike, optional) — Path to a directory in which a downloaded pretrained model configuration should be cached if the standard cache should not be used.
  • from_tf (bool, optional, defaults to False) — Load the model weights from a TensorFlow checkpoint save file (see docstring of pretrained_model_name_or_path argument).
  • force_download (bool, optional, defaults to False) — Whether or not to force the (re-)download of the model weights and configuration files, overriding the cached versions if they exist.
  • resume_download (bool, optional, defaults to False) — Whether or not to delete incompletely received files. Will attempt to resume the download if such a file exists.
  • proxies (Dict[str, str], optional) — A dictionary of proxy servers to use by protocol or endpoint, e.g., {'http': 'foo.bar:3128', 'http://hostname': 'foo.bar:4012'}. The proxies are used on each request.
  • output_loading_info(bool, optional, defaults to False) — Whether ot not to also return a dictionary containing missing keys, unexpected keys and error messages.
  • local_files_only(bool, optional, defaults to False) — Whether or not to only look at local files (e.g., not try downloading the model).
  • revision (str, optional, defaults to "main") — The specific model version to use. It can be a branch name, a tag name, or a commit id, since we use a git-based system for storing models and other artifacts on huggingface.co, so revision can be any identifier allowed by git.
  • trust_remote_code (bool, optional, defaults to False) — Whether or not to allow for custom models defined on the Hub in their own modeling files. This option should only be set to True for repositories you trust and in which you have read the code, as it will execute code present on the Hub on your local machine.
  • code_revision (str, optional, defaults to "main") — The specific revision to use for the code on the Hub, if the code leaves in a different repository than the rest of the model. It can be a branch name, a tag name, or a commit id, since we use a git-based system for storing models and other artifacts on huggingface.co, so revision can be any identifier allowed by git.
  • kwargs (additional keyword arguments, optional) — Can be used to update the configuration object (after it being loaded) and initiate the model (e.g., output_attentions=True). Behaves differently depending on whether a config is provided or automatically loaded:

    • If a configuration is provided with config, **kwargs will be directly passed to the underlying model’s __init__ method (we assume all relevant updates to the configuration have already been done)
    • If a configuration is not provided, kwargs will be first passed to the configuration class initialization function (from_pretrained()). Each key of kwargs that corresponds to a configuration attribute will be used to override said attribute with the supplied kwargs value. Remaining keys that do not correspond to any configuration attribute will be passed to the underlying model’s __init__ function.

Instantiate one of the model classes of the library (with a document question answering head) from a pretrained model.

The model class to instantiate is selected based on the model_type property of the config object (either passed as an argument or loaded from pretrained_model_name_or_path if possible), or when it’s missing, by falling back to using pattern matching on pretrained_model_name_or_path:

  • layoutlm — LayoutLMForQuestionAnswering (LayoutLM model)
  • layoutlmv2 — LayoutLMv2ForQuestionAnswering (LayoutLMv2 model)
  • layoutlmv3 — LayoutLMv3ForQuestionAnswering (LayoutLMv3 model)

The model is set in evaluation mode by default using model.eval() (so for instance, dropout modules are deactivated). To train the model, you should first set it back in training mode with model.train()

Examples:

>>> from transformers import AutoConfig, AutoModelForDocumentQuestionAnswering

>>> # Download model and configuration from huggingface.co and cache.
>>> model = AutoModelForDocumentQuestionAnswering.from_pretrained("impira/layoutlm-document-qa", revision="52e01b3")

>>> # Update configuration during loading
>>> model = AutoModelForDocumentQuestionAnswering.from_pretrained("impira/layoutlm-document-qa", revision="52e01b3", output_attentions=True)
>>> model.config.output_attentions
True

>>> # Loading from a TF checkpoint file instead of a PyTorch model (slower)
>>> config = AutoConfig.from_pretrained("./tf_model/layoutlm_tf_model_config.json")
>>> model = AutoModelForDocumentQuestionAnswering.from_pretrained(
...     "./tf_model/layoutlm_tf_checkpoint.ckpt.index", from_tf=True, config=config
... )

TFAutoModelForDocumentQuestionAnswering

class transformers.TFAutoModelForDocumentQuestionAnswering

< >

( *args **kwargs )

This is a generic model class that will be instantiated as one of the model classes of the library (with a document question answering head) when created with the from_pretrained() class method or the from_config() class method.

This class cannot be instantiated directly using __init__() (throws an error).

from_config

< >

( **kwargs )

Parameters

  • config (PretrainedConfig) — The model class to instantiate is selected based on the configuration class:

    • LayoutLMConfig configuration class: TFLayoutLMForQuestionAnswering (LayoutLM model)
    • LayoutLMv3Config configuration class: TFLayoutLMv3ForQuestionAnswering (LayoutLMv3 model)
  • attn_implementation (str, optional) — The attention implementation to use in the model (if relevant). Can be any of "eager" (manual implementation of the attention), "sdpa" (using F.scaled_dot_product_attention), or "flash_attention_2" (using Dao-AILab/flash-attention). By default, if available, SDPA will be used for torch>=2.1.1. The default is otherwise the manual "eager" implementation.

Instantiates one of the model classes of the library (with a document question answering head) from a configuration.

Note: Loading a model from its configuration file does not load the model weights. It only affects the model’s configuration. Use from_pretrained() to load the model weights.

Examples:

>>> from transformers import AutoConfig, TFAutoModelForDocumentQuestionAnswering

>>> # Download configuration from huggingface.co and cache.
>>> config = AutoConfig.from_pretrained("impira/layoutlm-document-qa", revision="52e01b3")
>>> model = TFAutoModelForDocumentQuestionAnswering.from_config(config)

from_pretrained

< >

( *model_args **kwargs )

Parameters

  • pretrained_model_name_or_path (str or os.PathLike) — Can be either:

    • A string, the model id of a pretrained model hosted inside a model repo on huggingface.co.
    • A path to a directory containing model weights saved using save_pretrained(), e.g., ./my_model_directory/.
    • A path or url to a PyTorch state_dict save file (e.g, ./pt_model/pytorch_model.bin). In this case, from_pt should be set to True and a configuration object should be provided as config argument. This loading path is slower than converting the PyTorch model in a TensorFlow model using the provided conversion scripts and loading the TensorFlow model afterwards.
  • model_args (additional positional arguments, optional) — Will be passed along to the underlying model __init__() method.
  • config (PretrainedConfig, optional) — Configuration for the model to use instead of an automatically loaded configuration. Configuration can be automatically loaded when:

    • The model is a model provided by the library (loaded with the model id string of a pretrained model).
    • The model was saved using save_pretrained() and is reloaded by supplying the save directory.
    • The model is loaded by supplying a local directory as pretrained_model_name_or_path and a configuration JSON file named config.json is found in the directory.
  • cache_dir (str or os.PathLike, optional) — Path to a directory in which a downloaded pretrained model configuration should be cached if the standard cache should not be used.
  • from_pt (bool, optional, defaults to False) — Load the model weights from a PyTorch checkpoint save file (see docstring of pretrained_model_name_or_path argument).
  • force_download (bool, optional, defaults to False) — Whether or not to force the (re-)download of the model weights and configuration files, overriding the cached versions if they exist.
  • resume_download (bool, optional, defaults to False) — Whether or not to delete incompletely received files. Will attempt to resume the download if such a file exists.
  • proxies (Dict[str, str], optional) — A dictionary of proxy servers to use by protocol or endpoint, e.g., {'http': 'foo.bar:3128', 'http://hostname': 'foo.bar:4012'}. The proxies are used on each request.
  • output_loading_info(bool, optional, defaults to False) — Whether ot not to also return a dictionary containing missing keys, unexpected keys and error messages.
  • local_files_only(bool, optional, defaults to False) — Whether or not to only look at local files (e.g., not try downloading the model).
  • revision (str, optional, defaults to "main") — The specific model version to use. It can be a branch name, a tag name, or a commit id, since we use a git-based system for storing models and other artifacts on huggingface.co, so revision can be any identifier allowed by git.
  • trust_remote_code (bool, optional, defaults to False) — Whether or not to allow for custom models defined on the Hub in their own modeling files. This option should only be set to True for repositories you trust and in which you have read the code, as it will execute code present on the Hub on your local machine.
  • code_revision (str, optional, defaults to "main") — The specific revision to use for the code on the Hub, if the code leaves in a different repository than the rest of the model. It can be a branch name, a tag name, or a commit id, since we use a git-based system for storing models and other artifacts on huggingface.co, so revision can be any identifier allowed by git.
  • kwargs (additional keyword arguments, optional) — Can be used to update the configuration object (after it being loaded) and initiate the model (e.g., output_attentions=True). Behaves differently depending on whether a config is provided or automatically loaded:

    • If a configuration is provided with config, **kwargs will be directly passed to the underlying model’s __init__ method (we assume all relevant updates to the configuration have already been done)
    • If a configuration is not provided, kwargs will be first passed to the configuration class initialization function (from_pretrained()). Each key of kwargs that corresponds to a configuration attribute will be used to override said attribute with the supplied kwargs value. Remaining keys that do not correspond to any configuration attribute will be passed to the underlying model’s __init__ function.

Instantiate one of the model classes of the library (with a document question answering head) from a pretrained model.

The model class to instantiate is selected based on the model_type property of the config object (either passed as an argument or loaded from pretrained_model_name_or_path if possible), or when it’s missing, by falling back to using pattern matching on pretrained_model_name_or_path:

  • layoutlm — TFLayoutLMForQuestionAnswering (LayoutLM model)
  • layoutlmv3 — TFLayoutLMv3ForQuestionAnswering (LayoutLMv3 model)

Examples:

>>> from transformers import AutoConfig, TFAutoModelForDocumentQuestionAnswering

>>> # Download model and configuration from huggingface.co and cache.
>>> model = TFAutoModelForDocumentQuestionAnswering.from_pretrained("impira/layoutlm-document-qa", revision="52e01b3")

>>> # Update configuration during loading
>>> model = TFAutoModelForDocumentQuestionAnswering.from_pretrained("impira/layoutlm-document-qa", revision="52e01b3", output_attentions=True)
>>> model.config.output_attentions
True

>>> # Loading from a PyTorch checkpoint file instead of a TensorFlow model (slower)
>>> config = AutoConfig.from_pretrained("./pt_model/layoutlm_pt_model_config.json")
>>> model = TFAutoModelForDocumentQuestionAnswering.from_pretrained(
...     "./pt_model/layoutlm_pytorch_model.bin", from_pt=True, config=config
... )

AutoModelForVisualQuestionAnswering

class transformers.AutoModelForVisualQuestionAnswering

< >

( *args **kwargs )

This is a generic model class that will be instantiated as one of the model classes of the library (with a visual question answering head) when created with the from_pretrained() class method or the from_config() class method.

This class cannot be instantiated directly using __init__() (throws an error).

from_config

< >

( **kwargs )

Parameters

Instantiates one of the model classes of the library (with a visual question answering head) from a configuration.

Note: Loading a model from its configuration file does not load the model weights. It only affects the model’s configuration. Use from_pretrained() to load the model weights.

Examples:

>>> from transformers import AutoConfig, AutoModelForVisualQuestionAnswering

>>> # Download configuration from huggingface.co and cache.
>>> config = AutoConfig.from_pretrained("dandelin/vilt-b32-finetuned-vqa")
>>> model = AutoModelForVisualQuestionAnswering.from_config(config)

from_pretrained

< >

( *model_args **kwargs )

Parameters

  • pretrained_model_name_or_path (str or os.PathLike) — Can be either:

    • A string, the model id of a pretrained model hosted inside a model repo on huggingface.co.
    • A path to a directory containing model weights saved using save_pretrained(), e.g., ./my_model_directory/.
    • A path or url to a tensorflow index checkpoint file (e.g, ./tf_model/model.ckpt.index). In this case, from_tf should be set to True and a configuration object should be provided as config argument. This loading path is slower than converting the TensorFlow checkpoint in a PyTorch model using the provided conversion scripts and loading the PyTorch model afterwards.
  • model_args (additional positional arguments, optional) — Will be passed along to the underlying model __init__() method.
  • config (PretrainedConfig, optional) — Configuration for the model to use instead of an automatically loaded configuration. Configuration can be automatically loaded when:

    • The model is a model provided by the library (loaded with the model id string of a pretrained model).
    • The model was saved using save_pretrained() and is reloaded by supplying the save directory.
    • The model is loaded by supplying a local directory as pretrained_model_name_or_path and a configuration JSON file named config.json is found in the directory.
  • state_dict (Dict[str, torch.Tensor], optional) — A state dictionary to use instead of a state dictionary loaded from saved weights file.

    This option can be used if you want to create a model from a pretrained configuration but load your own weights. In this case though, you should check if using save_pretrained() and from_pretrained() is not a simpler option.

  • cache_dir (str or os.PathLike, optional) — Path to a directory in which a downloaded pretrained model configuration should be cached if the standard cache should not be used.
  • from_tf (bool, optional, defaults to False) — Load the model weights from a TensorFlow checkpoint save file (see docstring of pretrained_model_name_or_path argument).
  • force_download (bool, optional, defaults to False) — Whether or not to force the (re-)download of the model weights and configuration files, overriding the cached versions if they exist.
  • resume_download (bool, optional, defaults to False) — Whether or not to delete incompletely received files. Will attempt to resume the download if such a file exists.
  • proxies (Dict[str, str], optional) — A dictionary of proxy servers to use by protocol or endpoint, e.g., {'http': 'foo.bar:3128', 'http://hostname': 'foo.bar:4012'}. The proxies are used on each request.
  • output_loading_info(bool, optional, defaults to False) — Whether ot not to also return a dictionary containing missing keys, unexpected keys and error messages.
  • local_files_only(bool, optional, defaults to False) — Whether or not to only look at local files (e.g., not try downloading the model).
  • revision (str, optional, defaults to "main") — The specific model version to use. It can be a branch name, a tag name, or a commit id, since we use a git-based system for storing models and other artifacts on huggingface.co, so revision can be any identifier allowed by git.
  • trust_remote_code (bool, optional, defaults to False) — Whether or not to allow for custom models defined on the Hub in their own modeling files. This option should only be set to True for repositories you trust and in which you have read the code, as it will execute code present on the Hub on your local machine.
  • code_revision (str, optional, defaults to "main") — The specific revision to use for the code on the Hub, if the code leaves in a different repository than the rest of the model. It can be a branch name, a tag name, or a commit id, since we use a git-based system for storing models and other artifacts on huggingface.co, so revision can be any identifier allowed by git.
  • kwargs (additional keyword arguments, optional) — Can be used to update the configuration object (after it being loaded) and initiate the model (e.g., output_attentions=True). Behaves differently depending on whether a config is provided or automatically loaded:

    • If a configuration is provided with config, **kwargs will be directly passed to the underlying model’s __init__ method (we assume all relevant updates to the configuration have already been done)
    • If a configuration is not provided, kwargs will be first passed to the configuration class initialization function (from_pretrained()). Each key of kwargs that corresponds to a configuration attribute will be used to override said attribute with the supplied kwargs value. Remaining keys that do not correspond to any configuration attribute will be passed to the underlying model’s __init__ function.

Instantiate one of the model classes of the library (with a visual question answering head) from a pretrained model.

The model class to instantiate is selected based on the model_type property of the config object (either passed as an argument or loaded from pretrained_model_name_or_path if possible), or when it’s missing, by falling back to using pattern matching on pretrained_model_name_or_path:

The model is set in evaluation mode by default using model.eval() (so for instance, dropout modules are deactivated). To train the model, you should first set it back in training mode with model.train()

Examples:

>>> from transformers import AutoConfig, AutoModelForVisualQuestionAnswering

>>> # Download model and configuration from huggingface.co and cache.
>>> model = AutoModelForVisualQuestionAnswering.from_pretrained("dandelin/vilt-b32-finetuned-vqa")

>>> # Update configuration during loading
>>> model = AutoModelForVisualQuestionAnswering.from_pretrained("dandelin/vilt-b32-finetuned-vqa", output_attentions=True)
>>> model.config.output_attentions
True

>>> # Loading from a TF checkpoint file instead of a PyTorch model (slower)
>>> config = AutoConfig.from_pretrained("./tf_model/vilt_tf_model_config.json")
>>> model = AutoModelForVisualQuestionAnswering.from_pretrained(
...     "./tf_model/vilt_tf_checkpoint.ckpt.index", from_tf=True, config=config
... )

AutoModelForVision2Seq

class transformers.AutoModelForVision2Seq

< >

( *args **kwargs )

This is a generic model class that will be instantiated as one of the model classes of the library (with a vision-to-text modeling head) when created with the from_pretrained() class method or the from_config() class method.

This class cannot be instantiated directly using __init__() (throws an error).

from_config

< >

( **kwargs )

Parameters

  • config (PretrainedConfig) — The model class to instantiate is selected based on the configuration class:

    • Blip2Config configuration class: Blip2ForConditionalGeneration (BLIP-2 model)
    • BlipConfig configuration class: BlipForConditionalGeneration (BLIP model)
    • GitConfig configuration class: GitForCausalLM (GIT model)
    • Idefics2Config configuration class: Idefics2ForConditionalGeneration (Idefics2 model)
    • InstructBlipConfig configuration class: InstructBlipForConditionalGeneration (InstructBLIP model)
    • Kosmos2Config configuration class: Kosmos2ForConditionalGeneration (KOSMOS-2 model)
    • LlavaConfig configuration class: LlavaForConditionalGeneration (LLaVa model)
    • LlavaNextConfig configuration class: LlavaNextForConditionalGeneration (LLaVA-NeXT model)
    • Pix2StructConfig configuration class: Pix2StructForConditionalGeneration (Pix2Struct model)
    • VipLlavaConfig configuration class: VipLlavaForConditionalGeneration (VipLlava model)
    • VisionEncoderDecoderConfig configuration class: VisionEncoderDecoderModel (Vision Encoder decoder model)
  • attn_implementation (str, optional) — The attention implementation to use in the model (if relevant). Can be any of "eager" (manual implementation of the attention), "sdpa" (using F.scaled_dot_product_attention), or "flash_attention_2" (using Dao-AILab/flash-attention). By default, if available, SDPA will be used for torch>=2.1.1. The default is otherwise the manual "eager" implementation.

Instantiates one of the model classes of the library (with a vision-to-text modeling head) from a configuration.

Note: Loading a model from its configuration file does not load the model weights. It only affects the model’s configuration. Use from_pretrained() to load the model weights.

Examples:

>>> from transformers import AutoConfig, AutoModelForVision2Seq

>>> # Download configuration from huggingface.co and cache.
>>> config = AutoConfig.from_pretrained("google-bert/bert-base-cased")
>>> model = AutoModelForVision2Seq.from_config(config)

from_pretrained

< >

( *model_args **kwargs )

Parameters

  • pretrained_model_name_or_path (str or os.PathLike) — Can be either:

    • A string, the model id of a pretrained model hosted inside a model repo on huggingface.co.
    • A path to a directory containing model weights saved using save_pretrained(), e.g., ./my_model_directory/.
    • A path or url to a tensorflow index checkpoint file (e.g, ./tf_model/model.ckpt.index). In this case, from_tf should be set to True and a configuration object should be provided as config argument. This loading path is slower than converting the TensorFlow checkpoint in a PyTorch model using the provided conversion scripts and loading the PyTorch model afterwards.
  • model_args (additional positional arguments, optional) — Will be passed along to the underlying model __init__() method.
  • config (PretrainedConfig, optional) — Configuration for the model to use instead of an automatically loaded configuration. Configuration can be automatically loaded when:

    • The model is a model provided by the library (loaded with the model id string of a pretrained model).
    • The model was saved using save_pretrained() and is reloaded by supplying the save directory.
    • The model is loaded by supplying a local directory as pretrained_model_name_or_path and a configuration JSON file named config.json is found in the directory.
  • state_dict (Dict[str, torch.Tensor], optional) — A state dictionary to use instead of a state dictionary loaded from saved weights file.

    This option can be used if you want to create a model from a pretrained configuration but load your own weights. In this case though, you should check if using save_pretrained() and from_pretrained() is not a simpler option.

  • cache_dir (str or os.PathLike, optional) — Path to a directory in which a downloaded pretrained model configuration should be cached if the standard cache should not be used.
  • from_tf (bool, optional, defaults to False) — Load the model weights from a TensorFlow checkpoint save file (see docstring of pretrained_model_name_or_path argument).
  • force_download (bool, optional, defaults to False) — Whether or not to force the (re-)download of the model weights and configuration files, overriding the cached versions if they exist.
  • resume_download (bool, optional, defaults to False) — Whether or not to delete incompletely received files. Will attempt to resume the download if such a file exists.
  • proxies (Dict[str, str], optional) — A dictionary of proxy servers to use by protocol or endpoint, e.g., {'http': 'foo.bar:3128', 'http://hostname': 'foo.bar:4012'}. The proxies are used on each request.
  • output_loading_info(bool, optional, defaults to False) — Whether ot not to also return a dictionary containing missing keys, unexpected keys and error messages.
  • local_files_only(bool, optional, defaults to False) — Whether or not to only look at local files (e.g., not try downloading the model).
  • revision (str, optional, defaults to "main") — The specific model version to use. It can be a branch name, a tag name, or a commit id, since we use a git-based system for storing models and other artifacts on huggingface.co, so revision can be any identifier allowed by git.
  • trust_remote_code (bool, optional, defaults to False) — Whether or not to allow for custom models defined on the Hub in their own modeling files. This option should only be set to True for repositories you trust and in which you have read the code, as it will execute code present on the Hub on your local machine.
  • code_revision (str, optional, defaults to "main") — The specific revision to use for the code on the Hub, if the code leaves in a different repository than the rest of the model. It can be a branch name, a tag name, or a commit id, since we use a git-based system for storing models and other artifacts on huggingface.co, so revision can be any identifier allowed by git.
  • kwargs (additional keyword arguments, optional) — Can be used to update the configuration object (after it being loaded) and initiate the model (e.g., output_attentions=True). Behaves differently depending on whether a config is provided or automatically loaded:

    • If a configuration is provided with config, **kwargs will be directly passed to the underlying model’s __init__ method (we assume all relevant updates to the configuration have already been done)
    • If a configuration is not provided, kwargs will be first passed to the configuration class initialization function (from_pretrained()). Each key of kwargs that corresponds to a configuration attribute will be used to override said attribute with the supplied kwargs value. Remaining keys that do not correspond to any configuration attribute will be passed to the underlying model’s __init__ function.

Instantiate one of the model classes of the library (with a vision-to-text modeling head) from a pretrained model.

The model class to instantiate is selected based on the model_type property of the config object (either passed as an argument or loaded from pretrained_model_name_or_path if possible), or when it’s missing, by falling back to using pattern matching on pretrained_model_name_or_path:

  • blip — BlipForConditionalGeneration (BLIP model)
  • blip-2 — Blip2ForConditionalGeneration (BLIP-2 model)
  • git — GitForCausalLM (GIT model)
  • idefics2 — Idefics2ForConditionalGeneration (Idefics2 model)
  • instructblip — InstructBlipForConditionalGeneration (InstructBLIP model)
  • kosmos-2 — Kosmos2ForConditionalGeneration (KOSMOS-2 model)
  • llava — LlavaForConditionalGeneration (LLaVa model)
  • llava_next — LlavaNextForConditionalGeneration (LLaVA-NeXT model)
  • pix2struct — Pix2StructForConditionalGeneration (Pix2Struct model)
  • vipllava — VipLlavaForConditionalGeneration (VipLlava model)
  • vision-encoder-decoder — VisionEncoderDecoderModel (Vision Encoder decoder model)

The model is set in evaluation mode by default using model.eval() (so for instance, dropout modules are deactivated). To train the model, you should first set it back in training mode with model.train()

Examples:

>>> from transformers import AutoConfig, AutoModelForVision2Seq

>>> # Download model and configuration from huggingface.co and cache.
>>> model = AutoModelForVision2Seq.from_pretrained("google-bert/bert-base-cased")

>>> # Update configuration during loading
>>> model = AutoModelForVision2Seq.from_pretrained("google-bert/bert-base-cased", output_attentions=True)
>>> model.config.output_attentions
True

>>> # Loading from a TF checkpoint file instead of a PyTorch model (slower)
>>> config = AutoConfig.from_pretrained("./tf_model/bert_tf_model_config.json")
>>> model = AutoModelForVision2Seq.from_pretrained(
...     "./tf_model/bert_tf_checkpoint.ckpt.index", from_tf=True, config=config
... )

TFAutoModelForVision2Seq

class transformers.TFAutoModelForVision2Seq

< >

( *args **kwargs )

This is a generic model class that will be instantiated as one of the model classes of the library (with a vision-to-text modeling head) when created with the from_pretrained() class method or the from_config() class method.

This class cannot be instantiated directly using __init__() (throws an error).

from_config

< >

( **kwargs )

Parameters

  • config (PretrainedConfig) — The model class to instantiate is selected based on the configuration class:
  • attn_implementation (str, optional) — The attention implementation to use in the model (if relevant). Can be any of "eager" (manual implementation of the attention), "sdpa" (using F.scaled_dot_product_attention), or "flash_attention_2" (using Dao-AILab/flash-attention). By default, if available, SDPA will be used for torch>=2.1.1. The default is otherwise the manual "eager" implementation.

Instantiates one of the model classes of the library (with a vision-to-text modeling head) from a configuration.

Note: Loading a model from its configuration file does not load the model weights. It only affects the model’s configuration. Use from_pretrained() to load the model weights.

Examples:

>>> from transformers import AutoConfig, TFAutoModelForVision2Seq

>>> # Download configuration from huggingface.co and cache.
>>> config = AutoConfig.from_pretrained("google-bert/bert-base-cased")
>>> model = TFAutoModelForVision2Seq.from_config(config)

from_pretrained

< >

( *model_args **kwargs )

Parameters

  • pretrained_model_name_or_path (str or os.PathLike) — Can be either:

    • A string, the model id of a pretrained model hosted inside a model repo on huggingface.co.
    • A path to a directory containing model weights saved using save_pretrained(), e.g., ./my_model_directory/.
    • A path or url to a PyTorch state_dict save file (e.g, ./pt_model/pytorch_model.bin). In this case, from_pt should be set to True and a configuration object should be provided as config argument. This loading path is slower than converting the PyTorch model in a TensorFlow model using the provided conversion scripts and loading the TensorFlow model afterwards.
  • model_args (additional positional arguments, optional) — Will be passed along to the underlying model __init__() method.
  • config (PretrainedConfig, optional) — Configuration for the model to use instead of an automatically loaded configuration. Configuration can be automatically loaded when:

    • The model is a model provided by the library (loaded with the model id string of a pretrained model).
    • The model was saved using save_pretrained() and is reloaded by supplying the save directory.
    • The model is loaded by supplying a local directory as pretrained_model_name_or_path and a configuration JSON file named config.json is found in the directory.
  • cache_dir (str or os.PathLike, optional) — Path to a directory in which a downloaded pretrained model configuration should be cached if the standard cache should not be used.
  • from_pt (bool, optional, defaults to False) — Load the model weights from a PyTorch checkpoint save file (see docstring of pretrained_model_name_or_path argument).
  • force_download (bool, optional, defaults to False) — Whether or not to force the (re-)download of the model weights and configuration files, overriding the cached versions if they exist.
  • resume_download (bool, optional, defaults to False) — Whether or not to delete incompletely received files. Will attempt to resume the download if such a file exists.
  • proxies (Dict[str, str], optional) — A dictionary of proxy servers to use by protocol or endpoint, e.g., {'http': 'foo.bar:3128', 'http://hostname': 'foo.bar:4012'}. The proxies are used on each request.
  • output_loading_info(bool, optional, defaults to False) — Whether ot not to also return a dictionary containing missing keys, unexpected keys and error messages.
  • local_files_only(bool, optional, defaults to False) — Whether or not to only look at local files (e.g., not try downloading the model).
  • revision (str, optional, defaults to "main") — The specific model version to use. It can be a branch name, a tag name, or a commit id, since we use a git-based system for storing models and other artifacts on huggingface.co, so revision can be any identifier allowed by git.
  • trust_remote_code (bool, optional, defaults to False) — Whether or not to allow for custom models defined on the Hub in their own modeling files. This option should only be set to True for repositories you trust and in which you have read the code, as it will execute code present on the Hub on your local machine.
  • code_revision (str, optional, defaults to "main") — The specific revision to use for the code on the Hub, if the code leaves in a different repository than the rest of the model. It can be a branch name, a tag name, or a commit id, since we use a git-based system for storing models and other artifacts on huggingface.co, so revision can be any identifier allowed by git.
  • kwargs (additional keyword arguments, optional) — Can be used to update the configuration object (after it being loaded) and initiate the model (e.g., output_attentions=True). Behaves differently depending on whether a config is provided or automatically loaded:

    • If a configuration is provided with config, **kwargs will be directly passed to the underlying model’s __init__ method (we assume all relevant updates to the configuration have already been done)
    • If a configuration is not provided, kwargs will be first passed to the configuration class initialization function (from_pretrained()). Each key of kwargs that corresponds to a configuration attribute will be used to override said attribute with the supplied kwargs value. Remaining keys that do not correspond to any configuration attribute will be passed to the underlying model’s __init__ function.

Instantiate one of the model classes of the library (with a vision-to-text modeling head) from a pretrained model.

The model class to instantiate is selected based on the model_type property of the config object (either passed as an argument or loaded from pretrained_model_name_or_path if possible), or when it’s missing, by falling back to using pattern matching on pretrained_model_name_or_path:

Examples:

>>> from transformers import AutoConfig, TFAutoModelForVision2Seq

>>> # Download model and configuration from huggingface.co and cache.
>>> model = TFAutoModelForVision2Seq.from_pretrained("google-bert/bert-base-cased")

>>> # Update configuration during loading
>>> model = TFAutoModelForVision2Seq.from_pretrained("google-bert/bert-base-cased", output_attentions=True)
>>> model.config.output_attentions
True

>>> # Loading from a PyTorch checkpoint file instead of a TensorFlow model (slower)
>>> config = AutoConfig.from_pretrained("./pt_model/bert_pt_model_config.json")
>>> model = TFAutoModelForVision2Seq.from_pretrained(
...     "./pt_model/bert_pytorch_model.bin", from_pt=True, config=config
... )

FlaxAutoModelForVision2Seq

class transformers.FlaxAutoModelForVision2Seq

< >

( *args **kwargs )

This is a generic model class that will be instantiated as one of the model classes of the library (with a vision-to-text modeling head) when created with the from_pretrained() class method or the from_config() class method.

This class cannot be instantiated directly using __init__() (throws an error).

from_config

< >

( **kwargs )

Parameters

  • config (PretrainedConfig) — The model class to instantiate is selected based on the configuration class:

    • VisionEncoderDecoderConfig configuration class: FlaxVisionEncoderDecoderModel (Vision Encoder decoder model)
  • attn_implementation (str, optional) — The attention implementation to use in the model (if relevant). Can be any of "eager" (manual implementation of the attention), "sdpa" (using F.scaled_dot_product_attention), or "flash_attention_2" (using Dao-AILab/flash-attention). By default, if available, SDPA will be used for torch>=2.1.1. The default is otherwise the manual "eager" implementation.

Instantiates one of the model classes of the library (with a vision-to-text modeling head) from a configuration.

Note: Loading a model from its configuration file does not load the model weights. It only affects the model’s configuration. Use from_pretrained() to load the model weights.

Examples:

>>> from transformers import AutoConfig, FlaxAutoModelForVision2Seq

>>> # Download configuration from huggingface.co and cache.
>>> config = AutoConfig.from_pretrained("google-bert/bert-base-cased")
>>> model = FlaxAutoModelForVision2Seq.from_config(config)

from_pretrained

< >

( *model_args **kwargs )

Parameters

  • pretrained_model_name_or_path (str or os.PathLike) — Can be either:

    • A string, the model id of a pretrained model hosted inside a model repo on huggingface.co.
    • A path to a directory containing model weights saved using save_pretrained(), e.g., ./my_model_directory/.
    • A path or url to a PyTorch state_dict save file (e.g, ./pt_model/pytorch_model.bin). In this case, from_pt should be set to True and a configuration object should be provided as config argument. This loading path is slower than converting the PyTorch model in a TensorFlow model using the provided conversion scripts and loading the TensorFlow model afterwards.
  • model_args (additional positional arguments, optional) — Will be passed along to the underlying model __init__() method.
  • config (PretrainedConfig, optional) — Configuration for the model to use instead of an automatically loaded configuration. Configuration can be automatically loaded when:

    • The model is a model provided by the library (loaded with the model id string of a pretrained model).
    • The model was saved using save_pretrained() and is reloaded by supplying the save directory.
    • The model is loaded by supplying a local directory as pretrained_model_name_or_path and a configuration JSON file named config.json is found in the directory.
  • cache_dir (str or os.PathLike, optional) — Path to a directory in which a downloaded pretrained model configuration should be cached if the standard cache should not be used.
  • from_pt (bool, optional, defaults to False) — Load the model weights from a PyTorch checkpoint save file (see docstring of pretrained_model_name_or_path argument).
  • force_download (bool, optional, defaults to False) — Whether or not to force the (re-)download of the model weights and configuration files, overriding the cached versions if they exist.
  • resume_download (bool, optional, defaults to False) — Whether or not to delete incompletely received files. Will attempt to resume the download if such a file exists.
  • proxies (Dict[str, str], optional) — A dictionary of proxy servers to use by protocol or endpoint, e.g., {'http': 'foo.bar:3128', 'http://hostname': 'foo.bar:4012'}. The proxies are used on each request.
  • output_loading_info(bool, optional, defaults to False) — Whether ot not to also return a dictionary containing missing keys, unexpected keys and error messages.
  • local_files_only(bool, optional, defaults to False) — Whether or not to only look at local files (e.g., not try downloading the model).
  • revision (str, optional, defaults to "main") — The specific model version to use. It can be a branch name, a tag name, or a commit id, since we use a git-based system for storing models and other artifacts on huggingface.co, so revision can be any identifier allowed by git.
  • trust_remote_code (bool, optional, defaults to False) — Whether or not to allow for custom models defined on the Hub in their own modeling files. This option should only be set to True for repositories you trust and in which you have read the code, as it will execute code present on the Hub on your local machine.
  • code_revision (str, optional, defaults to "main") — The specific revision to use for the code on the Hub, if the code leaves in a different repository than the rest of the model. It can be a branch name, a tag name, or a commit id, since we use a git-based system for storing models and other artifacts on huggingface.co, so revision can be any identifier allowed by git.
  • kwargs (additional keyword arguments, optional) — Can be used to update the configuration object (after it being loaded) and initiate the model (e.g., output_attentions=True). Behaves differently depending on whether a config is provided or automatically loaded:

    • If a configuration is provided with config, **kwargs will be directly passed to the underlying model’s __init__ method (we assume all relevant updates to the configuration have already been done)
    • If a configuration is not provided, kwargs will be first passed to the configuration class initialization function (from_pretrained()). Each key of kwargs that corresponds to a configuration attribute will be used to override said attribute with the supplied kwargs value. Remaining keys that do not correspond to any configuration attribute will be passed to the underlying model’s __init__ function.

Instantiate one of the model classes of the library (with a vision-to-text modeling head) from a pretrained model.

The model class to instantiate is selected based on the model_type property of the config object (either passed as an argument or loaded from pretrained_model_name_or_path if possible), or when it’s missing, by falling back to using pattern matching on pretrained_model_name_or_path:

  • vision-encoder-decoder — FlaxVisionEncoderDecoderModel (Vision Encoder decoder model)

Examples:

>>> from transformers import AutoConfig, FlaxAutoModelForVision2Seq

>>> # Download model and configuration from huggingface.co and cache.
>>> model = FlaxAutoModelForVision2Seq.from_pretrained("google-bert/bert-base-cased")

>>> # Update configuration during loading
>>> model = FlaxAutoModelForVision2Seq.from_pretrained("google-bert/bert-base-cased", output_attentions=True)
>>> model.config.output_attentions
True

>>> # Loading from a PyTorch checkpoint file instead of a TensorFlow model (slower)
>>> config = AutoConfig.from_pretrained("./pt_model/bert_pt_model_config.json")
>>> model = FlaxAutoModelForVision2Seq.from_pretrained(
...     "./pt_model/bert_pytorch_model.bin", from_pt=True, config=config
... )
< > Update on GitHub

←エージェントとツール コールバック→