Repository: huggingface/transformers
File: src/transformers/models/llama4/configuration_llama4.py (Python)

Code before:
        patch_size (`int`, *optional*, defaults to 14):
            The size (resolution) of each patch.
        norm_eps (`float`, *optional*, defaults to 1e-05):
            The epsilon used by the layer normalization layers.
        vision_feature_select_strategy (`str`, *optional*, defaults to `"default"`):
            The feature selection strategy used to select the vision features from the vision model.
            Should be same as in model's config. Can be one of `"default"` or `"full"`. If `"default"`, the CLS token is
            removed from the vision features.
        initializer_range (`float`, *optional*, defaults to 0.02):
            The standard deviation of the truncated_normal_initializer for initializing all weight matrices.
        pixel_shuffle_ratio (`float`, *optional*, defaults to 0.5):
            The ratio used for the pixel shuffle operation in the multi-modal projector. This controls the downsampling
            factor applied to image patches. The actual downsampling ratio is calculated as `1 / (pixel_shuffle_ratio^2)`.
            A value < 1.0 downsamples the visual features, reducing the number of tokens passed to the LLM.
        projector_input_dim (`int`, *optional*, defaults to 4096):
            Dimensionality of the input to the multi-modal projector.
        projector_output_dim (`int`, *optional*, defaults to 4096):
            Dimensionality of the output of the multi-modal projector.
        multi_modal_projector_bias (`bool`, *optional*, defaults to `False`):
            Whether to use bias in the multi-modal projector layers.
        projector_dropout (`float`, *optional*, defaults to 0.0):
            The dropout probability for the multi-modal projector. Increasing this value during training acts as
            regularization to prevent overfitting.
        attention_dropout (`float`, *optional*, defaults to 0.0):
            The dropout ratio for the attention probabilities. Similar to projector_dropout, increasing this value during
            training acts as regularization.
        rope_parameters (`RopeParameters`, *optional*):
            RoPE Parameters
    """

    base_model_tp_plan = {
        "model.layers.*.self_attn.q_proj": "colwise",
        "model.layers.*.self_attn.k_proj": "colwise",
        "model.layers.*.self_attn.v_proj": "colwise",
        "model.layers.*.self_attn.o_proj": "rowwise",
        "vision_adapter.mlp.fc1": "colwise",
        "vision_adapter.mlp.fc2": "rowwise",
        "patch_embedding.linear": "colwise_rep",
    }
    model_type = "llama4_vision_model"
    base_config_key = "vision_config"

    def __init__(
        self,
        hidden_size: int | None = 768,
        hidden_act: str | None = "gelu",
        num_hidden_layers: int | None = 34,
        num_attention_heads: int | None = 16,
        num_channels: int | None = 3,
        intermediate_size: int | None = 5632,
        vision_output_dim: int | None = 7680,

Review comment (suggestion):
```suggestion
        pixel_shuffle_ratio (`float`, *optional*, defaults to 0.5):
            Pixel-shuffle ratio for downsampling patch tokens. Smaller values produce fewer tokens (more downsampling).
        projector_input_dim (`int`, *optional*, defaults to 4096):
            Width of the vision adapter MLP before pixel shuffle. Larger value increases capacity and compute.
        projector_output_dim (`int`, *optional*, defaults to 4096):
            Output width of the vision adapter. Larger value yields higher-dimensional image features.
        multi_modal_projector_bias (`bool`, *optional*, defaults to `False`):
            Whether to use bias in the multi-modal projector layers.
        projector_dropout (`float`, *optional*, defaults to 0.0):
            Dropout rate inside the vision adapter MLP. Higher value adds more regularization.
        attention_dropout (`float`, *optional*, defaults to 0.0):
            Dropout rate on vision attention probabilities. Higher value adds more regularization.
```

Code after:
        patch_size (`int`, *optional*, defaults to 14):
            The size (resolution) of each patch.
        norm_eps (`float`, *optional*, defaults to 1e-05):
            The epsilon used by the layer normalization layers.
        vision_feature_select_strategy (`str`, *optional*, defaults to `"default"`):
            Controls which vision tokens are kept from the backbone. `"default"` drops the CLS token and `"full"` keeps all tokens.
        initializer_range (`float`, *optional*, defaults to 0.02):
            The standard deviation of the truncated_normal_initializer for initializing all weight matrices.
        pixel_shuffle_ratio (`float`, *optional*, defaults to 0.5):
            Pixel-shuffle ratio for downsampling patch tokens. Smaller values produce fewer tokens (more downsampling).
        projector_input_dim (`int`, *optional*, defaults to 4096):
            Width of the vision adapter MLP before pixel shuffle. Larger value increases capacity and compute.
        projector_output_dim (`int`, *optional*, defaults to 4096):
            Output width of the vision adapter. Larger value yields higher-dimensional image features.
        multi_modal_projector_bias (`bool`, *optional*, defaults to `False`):
            Whether to use bias in the multi-modal projector layers.
        projector_dropout (`float`, *optional*, defaults to 0.0):
            Dropout rate inside the vision adapter MLP. Higher value adds more regularization.
        attention_dropout (`float`, *optional*, defaults to 0.0):
            Dropout rate on vision attention probabilities. Higher value adds more regularization.
        rope_parameters (`RopeParameters`, *optional*):
            RoPE Parameters
    """

    base_model_tp_plan = {
        "model.layers.*.self_attn.q_proj": "colwise",
        "model.layers.*.self_attn.k_proj": "colwise",
        "model.layers.*.self_attn.v_proj": "colwise",
        "model.layers.*.self_attn.o_proj": "rowwise",
        "vision_adapter.mlp.fc1": "colwise",
        "vision_adapter.mlp.fc2": "rowwise",
        "patch_embedding.linear": "colwise_gather_output",
    }
    model_type = "llama4_vision_model"
    base_config_key = "vision_config"

    def __init__(
        self,
        hidden_size: int | None = 768,
        hidden_act: str | None = "gelu",
        num_hidden_layers: int | None = 34,
        num_attention_heads: int | None = 16,
        num_channels: int | None = 3,
        intermediate_size: int | None = 5632,
        vision_output_dim: int | None = 7680,
        image_size: int | None = 448,
        patch_size: int | None = 14,
        norm_eps: float | None = 1e-5,
        vision_feature_select_strategy: str | None = "default",
        initializer_range: float | None = 0.02,
        pixel_shuffle_ratio: float | None = 0.5,