CLAP
Overview
The CLAP model was proposed in Large Scale Contrastive Language-Audio pretraining with feature fusion and keyword-to-caption augmentation by Yusong Wu, Ke Chen, Tianyu Zhang, Yuchen Hui, Taylor Berg-Kirkpatrick, Shlomo Dubnov.
CLAP (Contrastive Language-Audio Pretraining) is a neural network trained on a variety of (audio, text) pairs. It can be instructed in to predict the most relevant text snippet, given an audio, without directly optimizing for the task. The CLAP model uses a SWINTransformer to get audio features from a log-Mel spectrogram input, and a RoBERTa model to get text features. Both the text and audio features are then projected to a latent space with identical dimension. The dot product between the projected audio and text features is then used as a similar score.
The abstract from the paper is the following:
Contrastive learning has shown remarkable success in the field of multimodal representation learning. In this paper, we propose a pipeline of contrastive language-audio pretraining to develop an audio representation by combining audio data with natural language descriptions. To accomplish this target, we first release LAION-Audio-630K, a large collection of 633,526 audio-text pairs from different data sources. Second, we construct a contrastive language-audio pretraining model by considering different audio encoders and text encoders. We incorporate the feature fusion mechanism and keyword-to-caption augmentation into the model design to further enable the model to process audio inputs of variable lengths and enhance the performance. Third, we perform comprehensive experiments to evaluate our model across three tasks: text-to-audio retrieval, zero-shot audio classification, and supervised audio classification. The results demonstrate that our model achieves superior performance in text-to-audio retrieval task. In audio classification tasks, the model achieves state-of-the-art performance in the zeroshot setting and is able to obtain performance comparable to models’ results in the non-zero-shot setting. LAION-Audio-6
This model was contributed by Younes Belkada and Arthur Zucker . The original code can be found here.
ClapConfig
class transformers.ClapConfig
< source >( text_config = None audio_config = None logit_scale_init_value = 14.285714285714285 projection_dim = 512 projection_hidden_act = 'relu' initializer_factor = 1.0 **kwargs )
Parameters
- text_config (
dict, optional) — Dictionary of configuration options used to initialize ClapTextConfig. - audio_config (
dict, optional) — Dictionary of configuration options used to initialize ClapAudioConfig. - logit_scale_init_value (
float, optional, defaults to 14.29) — The initial value of the logit_scale parameter. Default is used as per the original CLAP implementation. - projection_dim (
int, optional, defaults to 512) — Dimensionality of text and audio projection layers. - projection_hidden_act (
str, optional, defaults to"relu") — Activation function for the projection layers. - initializer_factor (
float, optional, defaults to 1.0) — Factor to scale the initialization of the model weights. - kwargs (optional) — Dictionary of keyword arguments.
ClapConfig is the configuration class to store the configuration of a ClapModel. It is used to instantiate a CLAP model according to the specified arguments, defining the text model and audio model configs. Instantiating a configuration with the defaults will yield a similar configuration to that of the CLAP laion/clap-htsat-fused architecture.
Configuration objects inherit from PretrainedConfig and can be used to control the model outputs. Read the documentation from PretrainedConfig for more information.
Example:
>>> from transformers import ClapConfig, ClapModel
>>> # Initializing a ClapConfig with laion-ai/base style configuration
>>> configuration = ClapConfig()
>>> # Initializing a ClapModel (with random weights) from the laion-ai/base style configuration
>>> model = ClapModel(configuration)
>>> # Accessing the model configuration
>>> configuration = model.config
>>> # We can also initialize a ClapConfig from a ClapTextConfig and a ClapAudioConfig
>>> from transformers import ClapTextConfig, ClapAudioConfig
>>> # Initializing a ClapText and ClapAudioConfig configuration
>>> config_text = ClapTextConfig()
>>> config_audio = ClapAudioConfig()
>>> config = ClapConfig.from_text_audio_configs(config_text, config_audio)from_text_audio_configs
< source >( text_config: ClapTextConfig audio_config: ClapAudioConfig **kwargs ) → ClapConfig
Instantiate a ClapConfig (or a derived class) from clap text model configuration and clap audio model configuration.
ClapTextConfig
class transformers.ClapTextConfig
< source >( vocab_size = 50265 hidden_size = 768 num_hidden_layers = 12 num_attention_heads = 12 intermediate_size = 3072 hidden_act = 'gelu' hidden_dropout_prob = 0.1 attention_probs_dropout_prob = 0.1 max_position_embeddings = 514 type_vocab_size = 1 initializer_factor = 1.0 layer_norm_eps = 1e-12 projection_dim = 512 pad_token_id = 1 bos_token_id = 0 eos_token_id = 2 position_embedding_type = 'absolute' use_cache = True projection_hidden_act = 'relu' **kwargs )
Parameters
- vocab_size (
int, optional, defaults to 30522) — Vocabulary size of the CLAP model. Defines the number of different tokens that can be represented by theinputs_idspassed when calling ClapTextModel. - hidden_size (
int, optional, defaults to 768) — Dimensionality of the encoder layers and the pooler layer. - num_hidden_layers (
int, optional, defaults to 12) — Number of hidden layers in the Transformer encoder. - num_attention_heads (
int, optional, defaults to 12) — Number of attention heads for each attention layer in the Transformer encoder. - intermediate_size (
int, optional, defaults to 3072) — Dimensionality of the “intermediate” (often named feed-forward) layer in the Transformer encoder. - hidden_act (
strorCallable, optional, defaults to"relu") — The non-linear activation function (function or string) in the encoder and pooler. If string,"relu","relu","silu"and"relu_new"are supported. - hidden_dropout_prob (
float, optional, defaults to 0.1) — The dropout probability for all fully connected layers in the embeddings, encoder, and pooler. - attention_probs_dropout_prob (
float, optional, defaults to 0.1) — The dropout ratio for the attention probabilities. - max_position_embeddings (
int, optional, defaults to 512) — The maximum sequence length that this model might ever be used with. Typically set this to something large just in case (e.g., 512 or 1024 or 2048). - type_vocab_size (
int, optional, defaults to 2) — The vocabulary size of thetoken_type_idspassed when calling ClapTextModel. - layer_norm_eps (
float, optional, defaults to 1e-12) — The epsilon used by the layer normalization layers. - position_embedding_type (
str, optional, defaults to"absolute") — Type of position embedding. Choose one of"absolute","relative_key","relative_key_query". For positional embeddings use"absolute". For more information on"relative_key", please refer to Self-Attention with Relative Position Representations (Shaw et al.). For more information on"relative_key_query", please refer to Method 4 in Improve Transformer Models with Better Relative Position Embeddings (Huang et al.). - is_decoder (
bool, optional, defaults toFalse) — Whether the model is used as a decoder or not. IfFalse, the model is used as an encoder. - use_cache (
bool, optional, defaults toTrue) — Whether or not the model should return the last key/values attentions (not used by all models). Only relevant ifconfig.is_decoder=True. - projection_hidden_act (
str, optional, defaults to"relu") — The non-linear activation function (function or string) in the projection layer. If string,"gelu","relu","silu"and"gelu_new"are supported. - projection_dim (
int, optional, defaults to 512) — Dimension of the projection head of theClapTextModelWithProjection.
This is the configuration class to store the configuration of a ClapTextModel. It is used to instantiate a CLAP model according to the specified arguments, defining the model architecture. Instantiating a configuration with the defaults will yield a similar configuration to that of the CLAP calp-hsat-fused architecture.
Configuration objects inherit from PretrainedConfig and can be used to control the model outputs. Read the documentation from PretrainedConfig for more information.
Examples:
>>> from transformers import ClapTextConfig, ClapTextModel
>>> # Initializing a CLAP text configuration
>>> configuration = ClapTextConfig()
>>> # Initializing a model (with random weights) from the configuration
>>> model = ClapTextModel(configuration)
>>> # Accessing the model configuration
>>> configuration = model.configClapAudioConfig
class transformers.ClapAudioConfig
< source >( window_size = 8 num_mel_bins = 64 spec_size = 256 hidden_act = 'gelu' patch_size = 4 patch_stride = [4, 4] num_classes = 527 hidden_size = 768 projection_dim = 512 depths = [2, 2, 6, 2] num_attention_heads = [4, 8, 16, 32] enable_fusion = False hidden_dropout_prob = 0.1 fusion_type = None patch_embed_input_channels = 1 flatten_patch_embeds = True patch_embeds_hidden_size = 96 enable_patch_layer_norm = True drop_path_rate = 0.0 attention_probs_dropout_prob = 0.0 qkv_bias = True mlp_ratio = 4.0 aff_block_r = 4 num_hidden_layers = 4 projection_hidden_act = 'relu' layer_norm_eps = 1e-05 initializer_factor = 1.0 **kwargs )
Parameters
- window_size (
int, optional, defaults to 8) — Image size of the spectrogram - num_mel_bins (
int, optional, defaults to 64) — Number of mel features used per frames. Should correspond to the value used in theClapProcessorclass. - spec_size (
int, optional, defaults to 256) — Desired input size of the spectrogram that the model supports. It can be different from the output of theClapFeatureExtractor, in which case the input features will be resized. Corresponds to theimage_sizeof the audio models. - hidden_act (
str, optional, defaults to"gelu") — The non-linear activation function (function or string) in the encoder and pooler. If string,"gelu","relu","silu"and"gelu_new"are supported. - patch_size (
int, optional, defaults to 4) — Patch size for the audio spectrogram - patch_stride (
list, optional, defaults to[4, 4]) — Patch stride for the audio spectrogram - num_classes (
int, optional, defaults to 527) — Number of classes used for the head training - hidden_size (
int, optional, defaults to 768) — Hidden size of the output of the audio encoder. Correspond to the dimension of the penultimate layer’s output,which is sent to the projection MLP layer. - projection_dim (
int, optional, defaults to 512) — Hidden size of the projection layer. - depths (
list, optional, defaults to[2, 2, 6, 2]) — Depths used for the Swin Layers of the audio model - num_attention_heads (
list, optional, defaults to[4, 8, 16, 32]) — Number of attention heads used for the Swin Layers of the audio model - enable_fusion (
bool, optional, defaults toFalse) — Whether or not to enable patch fusion. This is the main contribution of the authors, and should give the best results. - hidden_dropout_prob (
float, optional, defaults to 0.1) — The dropout probability for all fully connected layers in the encoder. - fusion_type (
[type], optional) — Fusion type used for the patch fusion. - patch_embed_input_channels (
int, optional, defaults to 1) — Number of channels used for the input spectrogram - flatten_patch_embeds (
bool, optional, defaults toTrue) — Whether or not to flatten the patch embeddings - patch_embeds_hidden_size (
int, optional, defaults to 96) — Hidden size of the patch embeddings. It is used as the number of output channels. - enable_patch_layer_norm (
bool, optional, defaults toTrue) — Whether or not to enable layer normalization for the patch embeddings - drop_path_rate (
float, optional, defaults to 0.0) — Drop path rate for the patch fusion - attention_probs_dropout_prob (
float, optional, defaults to 0.0) — The dropout ratio for the attention probabilities. - qkv_bias (
bool, optional, defaults toTrue) — Whether or not to add a bias to the query, key, value projections. - mlp_ratio (
float, optional, defaults to 4.0) — Ratio of the mlp hidden dim to embedding dim. - aff_block_r (
int, optional, defaults to 4) — downsize_ratio used in the AudioFF block - num_hidden_layers (
int, optional, defaults to 4) — Number of hidden layers in the Transformer encoder. - projection_hidden_act (
str, optional, defaults to"relu") — The non-linear activation function (function or string) in the projection layer. If string,"gelu","relu","silu"and"gelu_new"are supported. - layer_norm_eps (
[type], optional, defaults to 1e-05) — The epsilon used by the layer normalization layers. - initializer_factor (
float, optional, defaults to 1.0) — A factor for initializing all weight matrices (should be kept to 1, used internally for initialization testing).
This is the configuration class to store the configuration of a ClapAudioModel. It is used to instantiate a CLAP audio encoder according to the specified arguments, defining the model architecture. Instantiating a configuration with the defaults will yield a similar configuration to that of the audio encoder of the CLAP laion/clap-htsat-fused architecture.
Configuration objects inherit from PretrainedConfig and can be used to control the model outputs. Read the documentation from PretrainedConfig for more information.
Example:
>>> from transformers import ClapAudioConfig, ClapAudioModel
>>> # Initializing a ClapAudioConfig with laion/clap-htsat-fused style configuration
>>> configuration = ClapAudioConfig()
>>> # Initializing a ClapAudioModel (with random weights) from the laion/clap-htsat-fused style configuration
>>> model = ClapAudioModel(configuration)
>>> # Accessing the model configuration
>>> configuration = model.configClapFeatureExtractor
class transformers.ClapFeatureExtractor
< source >( feature_size = 64 sampling_rate = 48000 hop_length = 480 max_length_s = 10 fft_window_size = 1024 padding_value = 0.0 return_attention_mask = False frequency_min: float = 0 frequency_max: float = 14000 top_db: int = None truncation: str = 'fusion' padding: str = 'repeatpad' **kwargs )
Parameters
- feature_size (
int, optional, defaults to 64) — The feature dimension of the extracted Mel spectrograms. This corresponds to the number of mel filters (n_mels). - sampling_rate (
int, optional, defaults to 48000) — The sampling rate at which the audio files should be digitalized expressed in hertz (Hz). This only serves to warn users if the audio fed to the feature extractor does not have the same sampling rate. - hop_length (
int,optional, defaults to 480) — Length of the overlaping windows for the STFT used to obtain the Mel Spectrogram. The audio will be split in smallerframeswith a step ofhop_lengthbetween each frame. - max_length_s (
int, optional, defaults to 10) — The maximum input length of the model in seconds. This is used to pad the audio. - fft_window_size (
int, optional, defaults to 1024) — Size of the window (in samples) on which the Fourier transform is applied. This controls the frequency resolution of the spectrogram. 400 means that the fourrier transform is computed on windows of 400 samples. - padding_value (
float, optional, defaults to 0.0) — Padding value used to pad the audio. Should correspond to silences. - return_attention_mask (
bool, optional, defaults toFalse) — Whether or not the model should return the attention masks coresponding to the input. - frequency_min (
float, optional, defaults to 0) — The lowest frequency of interest. The STFT will not be computed for values below this. - frequency_max (
float, optional, defaults to 14000) — The highest frequency of interest. The STFT will not be computed for values above this. - top_db (
float, optional) — The highest decibel value used to convert the mel spectrogram to the log scale. For more details see theaudio_utils.power_to_dbfunction - truncation (
str, optional, defaults to"fusion") — Truncation pattern for long audio inputs. Two patterns are available:fusionwill use_random_mel_fusion, which stacks 3 random crops from the mel spectrogram and a downsampled version of the entire mel spectrogram. Ifconfig.fusionis set to True, shorter audios also need to to return 4 mels, which will just be a copy of the original mel obtained from the padded audio.rand_truncwill select a random crop of the mel spectrogram.
- padding (
str, optional, defaults to"repeatpad") — Padding pattern for shorter audio inputs. Three patterns were originally implemented:repeatpad: the audio is repeated, and then padded to fit themax_length.repeat: the audio is repeated and then cut to fit themax_lengthpad: the audio is padded.
Constructs a CLAP feature extractor.
This feature extractor inherits from SequenceFeatureExtractor which contains most of the main methods. Users should refer to this superclass for more information regarding those methods.
This class extracts mel-filter bank features from raw speech using a custom numpy implementation of the Short Time
Fourier Transform (STFT) which should match pytorch’s torch.stft equivalent.
to_dict
< source >( ) → Dict[str, Any]
Returns
Dict[str, Any]
Dictionary of all the attributes that make up this configuration instance, excpet for the mel filter banks, which do not need to be saved or printed as they are too long.
Serializes this instance to a Python dictionary.
ClapProcessor
class transformers.ClapProcessor
< source >( feature_extractor tokenizer )
Parameters
- feature_extractor (ClapFeatureExtractor) — The audio processor is a required input.
- tokenizer (RobertaTokenizerFast) — The tokenizer is a required input.
Constructs a CLAP processor which wraps a CLAP feature extractor and a RoBerta tokenizer into a single processor.
ClapProcessor offers all the functionalities of ClapFeatureExtractor and RobertaTokenizerFast. See the
__call__() and decode() for more information.
This method forwards all its arguments to RobertaTokenizerFast’s batch_decode(). Please refer to the docstring of this method for more information.
This method forwards all its arguments to RobertaTokenizerFast’s decode(). Please refer to the docstring of this method for more information.
ClapModel
class transformers.ClapModel
< source >( config: ClapConfig )
Parameters
- config (ClapConfig) — Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the from_pretrained() method to load the model weights.
This model inherits from PreTrainedModel. Check the superclass documentation for the generic methods the library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads etc.)
This model is also a PyTorch torch.nn.Module subclass. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior.
forward
< source >( input_ids: Optional = None input_features: Optional = None is_longer: Optional = None attention_mask: Optional = None position_ids: Optional = None return_loss: Optional = None output_attentions: Optional = None output_hidden_states: Optional = None return_dict: Optional = None ) → transformers.models.clap.modeling_clap.ClapOutput or tuple(torch.FloatTensor)
Parameters
- input_ids (
torch.LongTensorof shape(batch_size, sequence_length)) — Indices of input sequence tokens in the vocabulary. Padding will be ignored by default should you provide it.Indices can be obtained using AutoTokenizer. See PreTrainedTokenizer.encode() and PreTrainedTokenizer.call() for details.
- attention_mask (
torch.Tensorof shape(batch_size, sequence_length), optional) — Mask to avoid performing attention on padding token indices. Mask values selected in[0, 1]:- 1 for tokens that are not masked,
- 0 for tokens that are masked.
- position_ids (
torch.LongTensorof shape(batch_size, sequence_length), optional) — Indices of positions of each input sequence tokens in the position embeddings. Selected in the range[0, config.max_position_embeddings - 1]. - input_features (
torch.FloatTensorof shape(batch_size, num_channels, height, width)) — Input audio features. This should be returnes by the ClapFeatureExtractor class that you can also retrieve from AutoFeatureExtractor. SeeClapFeatureExtractor.__call__()for details. - return_loss (
bool, optional) — Whether or not to return the contrastive loss. - output_attentions (
bool, optional) — Whether or not to return the attentions tensors of all attention layers. Seeattentionsunder returned tensors for more detail. - output_hidden_states (
bool, optional) — Whether or not to return the hidden states of all layers. Seehidden_statesunder returned tensors for more detail. - return_dict (
bool, optional) — Whether or not to return a ModelOutput instead of a plain tuple.
Returns
transformers.models.clap.modeling_clap.ClapOutput or tuple(torch.FloatTensor)
A transformers.models.clap.modeling_clap.ClapOutput or a tuple of
torch.FloatTensor (if return_dict=False is passed or when config.return_dict=False) comprising various
elements depending on the configuration (<class 'transformers.models.clap.configuration_clap.ClapConfig'>) and inputs.
- loss (
torch.FloatTensorof shape(1,), optional, returned whenreturn_lossisTrue) — Contrastive loss for audio-text similarity. - logits_per_audio:(
torch.FloatTensorof shape(audio_batch_size, text_batch_size)) — The scaled dot product scores betweenaudio_embedsandtext_embeds. This represents the audio-text similarity scores. - logits_per_text:(
torch.FloatTensorof shape(text_batch_size, audio_batch_size)) — The scaled dot product scores betweentext_embedsandaudio_embeds. This represents the text-audio similarity scores. - text_embeds(
torch.FloatTensorof shape(batch_size, output_dim) — The text embeddings obtained by applying the projection layer to the pooled output of ClapTextModel. - audio_embeds(
torch.FloatTensorof shape(batch_size, output_dim) — The audio embeddings obtained by applying the projection layer to the pooled output of ClapAudioModel. - text_model_output(
BaseModelOutputWithPooling): The output of the ClapTextModel. - audio_model_output(
BaseModelOutputWithPooling): The output of the ClapAudioModel.
The ClapModel forward method, overrides the __call__ special method.
Although the recipe for forward pass needs to be defined within this function, one should call the Module
instance afterwards instead of this since the former takes care of running the pre and post processing steps while
the latter silently ignores them.
Examples:
>>> from datasets import load_dataset
>>> from transformers import AutoProcessor, ClapModel
>>> dataset = load_dataset("hf-internal-testing/ashraq-esc50-1-dog-example")
>>> audio_sample = dataset["train"]["audio"][0]["array"]
>>> model = ClapModel.from_pretrained("laion/clap-htsat-unfused")
>>> processor = AutoProcessor.from_pretrained("laion/clap-htsat-unfused")
>>> input_text = ["Sound of a dog", "Sound of vaccum cleaner"]
>>> inputs = processor(text=input_text, audios=audio_sample, return_tensors="pt", padding=True)
>>> outputs = model(**inputs)
>>> logits_per_audio = outputs.logits_per_audio # this is the audio-text similarity score
>>> probs = logits_per_audio.softmax(dim=-1) # we can take the softmax to get the label probabilitiesget_text_features
< source >( input_ids: Optional = None attention_mask: Optional = None position_ids: Optional = None output_attentions: Optional = None output_hidden_states: Optional = None return_dict: Optional = None ) → text_features (torch.FloatTensor of shape (batch_size, output_dim)
Parameters
- input_ids (
torch.LongTensorof shape(batch_size, sequence_length)) — Indices of input sequence tokens in the vocabulary. Padding will be ignored by default should you provide it.Indices can be obtained using AutoTokenizer. See PreTrainedTokenizer.encode() and PreTrainedTokenizer.call() for details.
- attention_mask (
torch.Tensorof shape(batch_size, sequence_length), optional) — Mask to avoid performing attention on padding token indices. Mask values selected in[0, 1]:- 1 for tokens that are not masked,
- 0 for tokens that are masked.
- position_ids (
torch.LongTensorof shape(batch_size, sequence_length), optional) — Indices of positions of each input sequence tokens in the position embeddings. Selected in the range[0, config.max_position_embeddings - 1]. - output_attentions (
bool, optional) — Whether or not to return the attentions tensors of all attention layers. Seeattentionsunder returned tensors for more detail. - output_hidden_states (
bool, optional) — Whether or not to return the hidden states of all layers. Seehidden_statesunder returned tensors for more detail. - return_dict (
bool, optional) — Whether or not to return a ModelOutput instead of a plain tuple.
Returns
text_features (torch.FloatTensor of shape (batch_size, output_dim)
The text embeddings obtained by applying the projection layer to the pooled output of ClapTextModel.
The ClapModel forward method, overrides the __call__ special method.
Although the recipe for forward pass needs to be defined within this function, one should call the Module
instance afterwards instead of this since the former takes care of running the pre and post processing steps while
the latter silently ignores them.
Examples:
>>> from transformers import AutoTokenizer, ClapModel
>>> model = ClapModel.from_pretrained("laion/clap-htsat-unfused")
>>> tokenizer = AutoTokenizer.from_pretrained("laion/clap-htsat-unfused")
>>> inputs = tokenizer(["the sound of a cat", "the sound of a dog"], padding=True, return_tensors="pt")
>>> text_features = model.get_text_features(**inputs)get_audio_features
< source >( input_features: Optional = None is_longer: Optional = None attention_mask: Optional = None output_attentions: Optional = None output_hidden_states: Optional = None return_dict: Optional = None ) → audio_features (torch.FloatTensor of shape (batch_size, output_dim)
Parameters
- input_features (
torch.FloatTensorof shape(batch_size, num_channels, height, width)) — Input audio features. This should be returnes by the ClapFeatureExtractor class that you can also retrieve from AutoFeatureExtractor. SeeClapFeatureExtractor.__call__()for details. - is_longer (
torch.FloatTensor, of shape(batch_size, 1), optional) — Whether the audio clip is longer thanmax_length. IfTrue, a feature fusion will be enabled to enhance the features. - output_attentions (
bool, optional) — Whether or not to return the attentions tensors of all attention layers. Seeattentionsunder returned tensors for more detail. - output_hidden_states (
bool, optional) — Whether or not to return the hidden states of all layers. Seehidden_statesunder returned tensors for more detail. - return_dict (
bool, optional) — Whether or not to return a ModelOutput instead of a plain tuple.
Returns
audio_features (torch.FloatTensor of shape (batch_size, output_dim)
The audio embeddings obtained by applying the projection layer to the pooled output of ClapAudioModel.
The ClapModel forward method, overrides the __call__ special method.
Although the recipe for forward pass needs to be defined within this function, one should call the Module
instance afterwards instead of this since the former takes care of running the pre and post processing steps while
the latter silently ignores them.
Examples:
>>> from transformers import AutoFeatureExtractor, ClapModel
>>> import torch
>>> model = ClapModel.from_pretrained("laion/clap-htsat-unfused")
>>> feature_extractor = AutoFeatureExtractor.from_pretrained("laion/clap-htsat-unfused")
>>> random_audio = torch.rand((16_000))
>>> inputs = feature_extractor(random_audio, return_tensors="pt")
>>> audio_features = model.get_audio_features(**inputs)ClapTextModel
The model can behave as an encoder (with only self-attention) as well as a decoder, in which case a layer of cross-attention is added between the self-attention layers, following the architecture described in Attention is all you need_ by Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser and Illia Polosukhin.
To behave as an decoder the model needs to be initialized with the is_decoder argument of the configuration set
to True. To be used in a Seq2Seq model, the model needs to initialized with both is_decoder argument and
add_cross_attention set to True; an encoder_hidden_states is then expected as an input to the forward pass.
.. _Attention is all you need: https://arxiv.org/abs/1706.03762
forward
< source >( input_ids: Optional = None attention_mask: Optional = None token_type_ids: Optional = None position_ids: Optional = None head_mask: Optional = None inputs_embeds: Optional = None encoder_hidden_states: Optional = None encoder_attention_mask: Optional = None past_key_values: Optional = None use_cache: Optional = None output_attentions: Optional = None output_hidden_states: Optional = None return_dict: Optional = None )
encoder_hidden_states (torch.FloatTensor of shape (batch_size, sequence_length, hidden_size), optional):
Sequence of hidden-states at the output of the last layer of the encoder. Used in the cross-attention if
the model is configured as a decoder.
encoder_attention_mask (torch.FloatTensor of shape (batch_size, sequence_length), optional):
Mask to avoid performing attention on the padding token indices of the encoder input. This mask is used in
the cross-attention if the model is configured as a decoder. Mask values selected in [0, 1]:
- 1 for tokens that are not masked,
- 0 for tokens that are masked.
past_key_values (
tuple(tuple(torch.FloatTensor))of lengthconfig.n_layerswith each tuple having 4 tensors of shape(batch_size, num_heads, sequence_length - 1, embed_size_per_head)): Contains precomputed key and value hidden states of the attention blocks. Can be used to speed up decoding.
If past_key_values are used, the user can optionally input only the last decoder_input_ids (those that
don’t have their past key value states given to this model) of shape (batch_size, 1) instead of all
decoder_input_ids of shape (batch_size, sequence_length).
use_cache (bool, optional):
If set to True, past_key_values key value states are returned and can be used to speed up decoding (see
past_key_values).
ClapTextModelWithProjection
class transformers.ClapTextModelWithProjection
< source >( config: ClapTextConfig )
Parameters
- config (ClapConfig) — Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the from_pretrained() method to load the model weights.
CLAP Text Model with a projection layer on top (a linear layer on top of the pooled output).
This model inherits from PreTrainedModel. Check the superclass documentation for the generic methods the library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads etc.)
This model is also a PyTorch torch.nn.Module subclass. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior.
forward
< source >( input_ids: Optional = None attention_mask: Optional = None position_ids: Optional = None output_attentions: Optional = None output_hidden_states: Optional = None return_dict: Optional = None ) → transformers.models.clap.modeling_clap.ClapTextModelOutput or tuple(torch.FloatTensor)
Parameters
- input_ids (
torch.LongTensorof shape(batch_size, sequence_length)) — Indices of input sequence tokens in the vocabulary. Padding will be ignored by default should you provide it.Indices can be obtained using AutoTokenizer. See PreTrainedTokenizer.encode() and PreTrainedTokenizer.call() for details.
- attention_mask (
torch.Tensorof shape(batch_size, sequence_length), optional) — Mask to avoid performing attention on padding token indices. Mask values selected in[0, 1]:- 1 for tokens that are not masked,
- 0 for tokens that are masked.
- position_ids (
torch.LongTensorof shape(batch_size, sequence_length), optional) — Indices of positions of each input sequence tokens in the position embeddings. Selected in the range[0, config.max_position_embeddings - 1]. - output_attentions (
bool, optional) — Whether or not to return the attentions tensors of all attention layers. Seeattentionsunder returned tensors for more detail. - output_hidden_states (
bool, optional) — Whether or not to return the hidden states of all layers. Seehidden_statesunder returned tensors for more detail. - return_dict (
bool, optional) — Whether or not to return a ModelOutput instead of a plain tuple.
Returns
transformers.models.clap.modeling_clap.ClapTextModelOutput or tuple(torch.FloatTensor)
A transformers.models.clap.modeling_clap.ClapTextModelOutput or a tuple of
torch.FloatTensor (if return_dict=False is passed or when config.return_dict=False) comprising various
elements depending on the configuration (<class 'transformers.models.clap.configuration_clap.ClapTextConfig'>) and inputs.
-
text_embeds (
torch.FloatTensorof shape(batch_size, output_dim)optional returned when model is initialized withwith_projection=True) — The text embeddings obtained by applying the projection layer to the pooler_output. -
last_hidden_state (
torch.FloatTensorof shape(batch_size, sequence_length, hidden_size)) — Sequence of hidden-states at the output of the last layer of the model. -
hidden_states (
tuple(torch.FloatTensor), optional, returned whenoutput_hidden_states=Trueis passed or whenconfig.output_hidden_states=True) — Tuple oftorch.FloatTensor(one for the output of the embeddings, if the model has an embedding layer, + one for the output of each layer) of shape(batch_size, sequence_length, hidden_size).Hidden-states of the model at the output of each layer plus the optional initial embedding outputs.
-
attentions (
tuple(torch.FloatTensor), optional, returned whenoutput_attentions=Trueis passed or whenconfig.output_attentions=True) — Tuple oftorch.FloatTensor(one for each layer) of shape(batch_size, num_heads, sequence_length, sequence_length).Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads.
The ClapTextModelWithProjection forward method, overrides the __call__ special method.
Although the recipe for forward pass needs to be defined within this function, one should call the Module
instance afterwards instead of this since the former takes care of running the pre and post processing steps while
the latter silently ignores them.
Examples:
>>> from transformers import AutoTokenizer, ClapTextModelWithProjection
>>> model = ClapTextModelWithProjection.from_pretrained("laion/clap-htsat-unfused")
>>> tokenizer = AutoTokenizer.from_pretrained("laion/clap-htsat-unfused")
>>> inputs = tokenizer(["a sound of a cat", "a sound of a dog"], padding=True, return_tensors="pt")
>>> outputs = model(**inputs)
>>> text_embeds = outputs.text_embedsClapAudioModel
forward
< source >( input_features: Optional = None is_longer: Optional = None output_attentions: Optional = None output_hidden_states: Optional = None return_dict: Optional = None ) → transformers.modeling_outputs.BaseModelOutputWithPooling or tuple(torch.FloatTensor)
Parameters
- input_features (
torch.FloatTensorof shape(batch_size, num_channels, height, width)) — Input audio features. This should be returnes by the ClapFeatureExtractor class that you can also retrieve from AutoFeatureExtractor. SeeClapFeatureExtractor.__call__()for details. - is_longer (
torch.FloatTensor, of shape(batch_size, 1), optional) — Whether the audio clip is longer thanmax_length. IfTrue, a feature fusion will be enabled to enhance the features. - output_attentions (
bool, optional) — Whether or not to return the attentions tensors of all attention layers. Seeattentionsunder returned tensors for more detail. - output_hidden_states (
bool, optional) — Whether or not to return the hidden states of all layers. Seehidden_statesunder returned tensors for more detail. - return_dict (
bool, optional) — Whether or not to return a ModelOutput instead of a plain tuple.
Returns
transformers.modeling_outputs.BaseModelOutputWithPooling or tuple(torch.FloatTensor)
A transformers.modeling_outputs.BaseModelOutputWithPooling or a tuple of
torch.FloatTensor (if return_dict=False is passed or when config.return_dict=False) comprising various
elements depending on the configuration (<class 'transformers.models.clap.configuration_clap.ClapAudioConfig'>) and inputs.
-
last_hidden_state (
torch.FloatTensorof shape(batch_size, sequence_length, hidden_size)) — Sequence of hidden-states at the output of the last layer of the model. -
pooler_output (
torch.FloatTensorof shape(batch_size, hidden_size)) — Last layer hidden-state of the first token of the sequence (classification token) after further processing through the layers used for the auxiliary pretraining task. E.g. for BERT-family of models, this returns the classification token after processing through a linear layer and a tanh activation function. The linear layer weights are trained from the next sentence prediction (classification) objective during pretraining. -
hidden_states (
tuple(torch.FloatTensor), optional, returned whenoutput_hidden_states=Trueis passed or whenconfig.output_hidden_states=True) — Tuple oftorch.FloatTensor(one for the output of the embeddings, if the model has an embedding layer, + one for the output of each layer) of shape(batch_size, sequence_length, hidden_size).Hidden-states of the model at the output of each layer plus the optional initial embedding outputs.
-
attentions (
tuple(torch.FloatTensor), optional, returned whenoutput_attentions=Trueis passed or whenconfig.output_attentions=True) — Tuple oftorch.FloatTensor(one for each layer) of shape(batch_size, num_heads, sequence_length, sequence_length).Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads.
The ClapAudioModel forward method, overrides the __call__ special method.
Although the recipe for forward pass needs to be defined within this function, one should call the Module
instance afterwards instead of this since the former takes care of running the pre and post processing steps while
the latter silently ignores them.
Examples:
>>> from datasets import load_dataset
>>> from transformers import AutoProcessor, ClapAudioModel
>>> dataset = load_dataset("hf-internal-testing/ashraq-esc50-1-dog-example")
>>> audio_sample = dataset["train"]["audio"][0]["array"]
>>> model = ClapAudioModel.from_pretrained("laion/clap-htsat-fused")
>>> processor = AutoProcessor.from_pretrained("laion/clap-htsat-fused")
>>> inputs = processor(audios=audio_sample, return_tensors="pt")
>>> outputs = model(**inputs)
>>> last_hidden_state = outputs.last_hidden_stateClapAudioModelWithProjection
class transformers.ClapAudioModelWithProjection
< source >( config: ClapAudioConfig )
Parameters
- config (ClapConfig) — Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the from_pretrained() method to load the model weights.
CLAP Audio Model with a projection layer on top (a linear layer on top of the pooled output).
This model inherits from PreTrainedModel. Check the superclass documentation for the generic methods the library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads etc.)
This model is also a PyTorch torch.nn.Module subclass. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior.
forward
< source >( input_features: Optional = None is_longer: Optional = None output_attentions: Optional = None output_hidden_states: Optional = None return_dict: Optional = None ) → transformers.models.clap.modeling_clap.ClapAudioModelOutput or tuple(torch.FloatTensor)
Parameters
- input_features (
torch.FloatTensorof shape(batch_size, num_channels, height, width)) — Input audio features. This should be returnes by the ClapFeatureExtractor class that you can also retrieve from AutoFeatureExtractor. SeeClapFeatureExtractor.__call__()for details. - is_longer (
torch.FloatTensor, of shape(batch_size, 1), optional) — Whether the audio clip is longer thanmax_length. IfTrue, a feature fusion will be enabled to enhance the features. - output_attentions (
bool, optional) — Whether or not to return the attentions tensors of all attention layers. Seeattentionsunder returned tensors for more detail. - output_hidden_states (
bool, optional) — Whether or not to return the hidden states of all layers. Seehidden_statesunder returned tensors for more detail. - return_dict (
bool, optional) — Whether or not to return a ModelOutput instead of a plain tuple.
Returns
transformers.models.clap.modeling_clap.ClapAudioModelOutput or tuple(torch.FloatTensor)
A transformers.models.clap.modeling_clap.ClapAudioModelOutput or a tuple of
torch.FloatTensor (if return_dict=False is passed or when config.return_dict=False) comprising various
elements depending on the configuration (<class 'transformers.models.clap.configuration_clap.ClapAudioConfig'>) and inputs.
-
audio_embeds (
torch.FloatTensorof shape(batch_size, hidden_size)) — The Audio embeddings obtained by applying the projection layer to the pooler_output. -
last_hidden_state (
torch.FloatTensorof shape(batch_size, sequence_length, hidden_size)) — Sequence of hidden-states at the output of the last layer of the model. -
attentions (
tuple(torch.FloatTensor), optional, returned whenoutput_attentions=Trueis passed or whenconfig.output_attentions=True) — Tuple oftorch.FloatTensor(one for each layer) of shape(batch_size, num_heads, sequence_length, sequence_length).Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads.
-
hidden_states (
tuple(torch.FloatTensor), optional, returned whenoutput_hidden_states=Trueis passed or whenconfig.output_hidden_states=True) — Tuple oftorch.FloatTensor(one for the output of the embeddings, if the model has an embedding layer, + one for the output of each layer) of shape(batch_size, sequence_length, hidden_size).Hidden-states of the model at the output of each layer plus the optional initial embedding outputs.
The ClapAudioModelWithProjection forward method, overrides the __call__ special method.
Although the recipe for forward pass needs to be defined within this function, one should call the Module
instance afterwards instead of this since the former takes care of running the pre and post processing steps while
the latter silently ignores them.
Examples:
>>> from datasets import load_dataset
>>> from transformers import ClapAudioModelWithProjection, ClapProcessor
>>> model = ClapAudioModelWithProjection.from_pretrained("laion/clap-htsat-fused")
>>> processor = ClapProcessor.from_pretrained("laion/clap-htsat-fused")
>>> dataset = load_dataset("hf-internal-testing/ashraq-esc50-1-dog-example")
>>> audio_sample = dataset["train"]["audio"][0]["array"]
>>> inputs = processor(audios=audio_sample, return_tensors="pt")
>>> outputs = model(**inputs)
>>> audio_embeds = outputs.audio_embeds