push

content/article.md

@@ -164,11 +164,37 @@ Semantics stay in the model (a Linear stays a Linear), distribution is orthogona
 
 
 ## <a id="layers-attentions-caches"></a> Layers, attentions and caches
-With th
+
+Following the same logic, the _nature_ of attention and caching per layer of a model should not be hardcoded. We should be able to specify in a configuration-based fashion how each layer is implemented. Thus we defined a mapping that can be then
+
+
+```python
+ALLOWED_LAYER_TYPES = (
+    "full_attention",
+    "sliding_attention",
+    "chunked_attention",
+    "linear_attention",
+    ...
+)
+```
+
+and the configuration can be _explicit_ about which attention type is in which layer, see e.g. gpt-oss, which alternates sliding and full attention:
+
+```python
+  "layer_types": [
+    "sliding_attention",
+    "full_attention",
+    ...,
+    "sliding_attention",
+    "full_attention"
+  ],
+```
+
+This is [minimal](#minimal-user-api) to implement on the user side, and allows to keep the modeling untouched. It is also [easy to tweak](#modular-toolbox).
 
 ## <a id="community-kernels"></a>Community Kernels
 
-The same principle extends to normalization, activation, and other hot paths. The model defines **semantics**; a kernel defines **how** to execute them faster. We annotate the module to borrow a community‑provided forward, keeping a [consistent public surface](#consistent-public-surface)
+The same principle extends to normalization, activation, and other code paths. The model defines **semantics**; a kernel defines **how** to execute them faster. We annotate the module to borrow a community‑provided forward, keeping a [consistent public surface](#consistent-public-surface)
 
 ```python
 @use_kernel_forward_from_hub("RMSNorm")
@@ -191,9 +217,10 @@ To get this graph, I used the heuristic of modular inheritance.
 2. In this `modular` file, what models, configurations and processings are imported?
 3. Recurse through the model list that way. 
 
-So what do we see? Llama is a basis for many models, and it shows. 
+So what do we see? Llama is a basis for many models, and it shows.
 Radically different architectures such as mamba have spawned their own dependency subgraph.
-{{{fragment-d3-graph}}}
+
+{{{fragment-dependency-graph}}}
 
 
  But there is no similar miracle for VLMs across the board.
@@ -204,7 +231,9 @@ One problem is, this is only for `modular` models. Several models do NOT have a
 
 ## Too many models, yet not enough, are alike
 
-So I looked into Jaccard similarity, which we use to measure set differences. I know that code is more than a set of characters stringed together, but it is a correct proxy for now. You can check out  [[find_dependencies.py]] . 
+So I looked into Jaccard similarity, which we use to measure set differences. I know that code is more than a set of characters stringed together, but it is a correct proxy for now. You can check out  [[find_dependencies.py]] .
+
+{{{fragment-model-timeline}}} 
 
 {{{fragment-terminal}}}
 
@@ -215,14 +244,14 @@ The yellow areas are places where models are very different to each other. We ca
 
 We don't have cookbook for common VLM patterns (image token scatter, multi‑tower encoders, cross‑attn bridges). This is one of the main improvement points where we can work.
 
-So initially I thought of abstracting away the mixing of `inputs_embeds`, the tensor fed into an llm decoder in 95% of the existing VLMs. It would have looked like something like 
+For instance, I thought of abstracting away the mixing of `inputs_embeds`, the tensor fed into an llm decoder in 95% of the existing VLMs. It would have looked like something like 
 
 ```python
 class InputsEmbeddingMixerMixin(nn.Module):
     #
 ```
 
-But this is breaking [Standardize, don't abstract](#standardize-dont-abstract). Embedding mixin is part of the model, removing it would break it. A user opening `modeling_qwen2.5_vl` should not have to go to another file. 
+But this is [abstracting away an important component of the modeling.](#standardize-dont-abstract). Embedding mixin is part of the model, removing it would break it. A user opening `modeling_qwen2.5_vl` should not have to go to another file. 
 
 This is the current state of abstractions across a modeling file:
 
@@ -279,7 +308,7 @@ But this is _within_ the modeling file, not in the `PreTrainedModel` base class.
 So the question abounds naturally: How can we modularize more?
 I took again a similarity measure and looked at the existing graphs. The tool is available on this [ZeroGPU-enabled Space](https://huggingface.co/spaces/Molbap/transformers-modular-refactor). It scans the whole transformers repository, and outputs a graph of candidates across models, using either a Jaccard similarity index (simple) or a SentenceTransformers embedding model. It is understandable that [encoder models still have a lion's share of the game.](#encoders-ftw) See also [Tom Aarsen and Arhur Bresnu's great blog post on the topic of sparse embeddings.](https://huggingface.co/blog/train-sparse-encoder).
 
-{{fragment-modular-growth}}
+{{{fragment-loc-growth}}}
 
 ## <a id="encoders-ftw"></a> The neverending stories of encoder models.
 
@@ -338,7 +367,7 @@ Adding a model to transformers means:
 - having it immediately available to the community
 - usable in vLLM, SGLang, and so on without additional code. In April 2025, transformers was added as a backend to run models on vLLM, which optimizes throughput/latency on top of existing transformers architectures [as seen in this great blog post.](https://blog.vllm.ai/2025/04/11/transformers-backend.html)
 
-This cements 
+This cements the need even more for a [consistent public surface](#consistent-public-surface): we are now a backend, and there's more optimized software than us to handle serving. At the time of writing, more effort is done in that direction. We already have compatible configs for VLMs for vLLM (say that three times fast), [here for GLM4 video support](https://github.com/huggingface/transformers/pull/40696/files),  
 
 ## Cooking faster CUDA warmups