Introduction
One million lines of python code. Through them, the transformers library supports more than 400 model architectures, from state-of-the-art LLMs and VLMs to specialized models for audio, video, and tables.
Built on PyTorch, it's a foundational tool for modern LLM usage, research, education, and tens of thousands of other open-source projects. Each AI model is added by the community, harmonized into a consistent interface, and tested daily on a CI to ensure reproducibility.
This scale presents a monumental engineering challenge.
How do you keep such a ship afloat, made of so many moving, unrelated parts, contributed to by a buzzing hivemind? Especially as the pace of ML research accelerates? We receive constant feedback on everything from function signatures with hundreds of arguments to duplicated code and optimization concerns, and we listen to all of it, or try to. The library's usage keeps on growing, and we are a small team of maintainers and contributors, backed by hundreds of open-source community members. We continue supporting all models that come out and will continue to do so in the foreseeable future.
This post dissects the design philosophy that makes this possible. It's a continuation of our older principles, detailed on our previous philosophy page, as well as its accompanying blog post from 2022. More recently, and I recommend the read if it's not done yet, a blog post about recent upgrades to transformers was written, explaining in particular what makes the library faster today. Again, all of that development was only made possible thanks to these principles.
We codify the "tenets" that guide our development, demonstrate how they are implemented in code, and show the measurable impact they have on the library's sustainability and growth.
For any OSS maintainer, power user, or contributor, this is the map to understanding, using, and building upon transformers, but not only: any project of comparable size will require you to make deep choices, not only on design and choice of abstraction, but on the very mindset of the software you are building.
Tenets exemplified will have their summary available on hover.
External links to articles will help you solidify your knowledge.
Several interactive visualisations are available as you go - scroll, zoom, drag away.
The core tenets of transformers
We summarize the foundations on which we've built everything, and write the "tenets" of the library. They behave like software interfaces, hence it is crucial that they are explicitly written down. However opinionated they are, they have evolved over time.
Note that the library evolved towards these principles, and that they emerged from decisions taken, and once emerged they were recognized as critical.
-
Source of Truth
We aim be a [source of truth for all model definitions](#https://huggingface.co/blog/transformers-model-definition). This is not a tenet, but something that still guides our decisions. Model implementations should be reliable, reproducible, and faithful to the original performances.
This overarching guideline ensures quality and reproducibility across all models in the library. -
One Model, One File
All inference and training core logic has to be visible, topโtoโbottom, to maximize each model's hackability.
Every model should be completely understandable and hackable by reading a single file from top to bottom. -
Code is Product
Optimize for reading, diffing, and tweaking, our users are power users. Variables can be explicit, full words, even several words, readability is primordial.
Code quality matters as much as functionality - optimize for human readers, not just computers. -
Standardize, Don't Abstract
If it's model behavior, keep it in the file; abstractions only for generic infra.
Model-specific logic belongs in the model file, not hidden behind abstractions. -
DRY* (DO Repeat Yourself)
Copy when it helps users; keep successors in sync without centralizing behavior.
Amendment: With the introduction and global adoption of modular transformers, we do not repeat any logic in the modular files, but end user files remain faithful to the original tenet.
Strategic duplication can improve readability and maintainability when done thoughtfully. -
Minimal User API
Config, model, preprocessing; from_pretrained, save_pretrained, push_to_hub. We want the least amount of codepaths. Reading should be obvious, configurations should be obvious.
Keep the public interface simple and predictable, users should know what to expect. -
Backwards Compatibility
Evolve by additive standardization, never break public APIs.
Any artifact that was once on the hub and loadable with transformers should be usable indefinitely with the same interface. Further, public methods should not change to avoid breaking dependencies. Once something is public, it stays public, evolution through addition, not breaking changes.
-
Consistent Public Surface
Same argument names, same outputs, hidden states and attentions exposed, enforced by tests. This is a goal we have as well as a tenet.
All models should feel familiar - consistent interfaces reduce cognitive load.
When a PR is merged, it is because the contribution is worthwhile, and that the transformers team finds the design of the contribution to be aligned with what is above.
Does all the code in the library follow strictly these tenets? No. The library is a gigantic house with connected nooks, corridors, crannies everywhere built by thousands of different workers. We try to make it so all the code added is compliant, because if we fail and merge it, we cannot change it lest we break backwards compatibility.
For instance, one function essential to the implementation of Rotary Positional Embeddings is identical in 70 modeling_<file>.py across src/transformers/models/. Why keep it? Because we want all the model logic to be contained in the modeling file. In order to do that, we do repeat ourselves.
def rotate_half(x):
"""Rotates half the hidden dims of the input."""
x1 = x[..., : x.shape[-1] // 2]
x2 = x[..., x.shape[-1] // 2 :]
return torch.cat((-x2, x1), dim=-1)
You can use a simple regex to look at all methods of a given name across your codebase and look at their differences and similarities, that's what I did (+ a hash to avoid quadraticity).
We want all models to have self-contained modeling code.
Every core functionality must be in the modeling code, every non-core functionality can be outside of it.
This comes as a great cost. Enter the #Copied from... mechanism: for a long time, these comments were indicating that some code was copied from another model, saving time both for the reviewers and for the CI. But the LOC count kept creeping up. Each new model copied over hundreds of lines that we considered largely boilerplate, yet, we could not remove them.
We needed to separate both principles that were so far intertwined, repetition and hackabilty.
What was the solution to this?
Modular transformers
Transformers is an opiniated library. The previous philosophy page, and the blog post were already pointing at the drawbacks mentioned just above, which have been iteratively addressed. modular transformers were introduced, allowing a form of inheritance without breaking One model, One file.
We amended the principle of DRY* by removing progressively all pieces of code that were "copied from" another file.
It works as follows. In order to contribute a model, say for instance define a modular_ file that can inherit from any function across all other modeling, configuration and processor files.
{{{fragment-glm-compare}}}
As you can see, we can now define any model as a modular of another.
You might think "well that's just how inheritance works". The crucial difference is that we do visibly what is essentially the compiler's job: by unrolling the inheritances, we make visible all of the modeling code, keeping it all in one piece.
What is the consequence? When adding a model, we do not need to go over the entire modeling file. The modular (left side above) is enough.
When AutoModel.from_pretrained(...) is called, it is indeed the modeling (right side) that is ran, and all the tests are run on the modeling code.
What does that gives us?
modular_*.py declares reuse; the expanded modeling file stays visible (tenet kept). Reviewers and contributors maintain the shard, not the repetition. Next: the measurable effect on effective LOC and maintenance cost.
A maintainable control surface
The effect of modular can be measured straight from git history: at every commit, we look under the model directory. If it only has a modeling file, we add its LOC count. However, if a model has a modular_*.py and a corresponding automatically generated modeling_*/.py, we only count the LOC under the modular file. The modeling code has no maintenance cost as it is strictly dependent on the modular file.
That gives an โeffective LOCโ curve: the ๐บ๐ฎ๐ถ๐ป๐๐ฒ๐ป๐ฎ๐ป๐ฐ๐ฒ ๐๐๐ฟ๐ณ๐ฎ๐ฐ๐ฒ.
๐๐๐๐ ๐น๐ผ๐ผ๐ธ ๐ฎ๐ ๐๐ต๐ฒ ๐ฟ๐ฒ๐๐๐น๐: ๐๐ต๐ฒ ๐ด๐ฟ๐ผ๐๐๐ต ๐ฟ๐ฎ๐๐ฒ ๐ผ๐ณ ๐น๐ถ๐ป๐ฒ๐ ๐ผ๐ณ ๐ฐ๐ผ๐ฑ๐ฒ ๐ฐ๐ผ๐น๐น๐ฎ๐ฝ๐๐ฒ๐ฑ! Counting raw ๐๐๐๐๐๐๐๐_*.๐๐ข (with โCopied fromโฆโ everywhere) we were around 362 new LOC/day; with ๐๐๐๐๐๐๐ in place the effective rate is ~25 LOC/day. About ๐ญ๐ฑร ๐น๐ผ๐๐ฒ๐ฟ! Had we continued with a strict "one model, one file" policy who knows where we'd have ended up.
Less code to hand-maintain means fewer places to break: cyclomatic complexity isnโt LOC, but they strongly correlate.
{{{fragment-loc-growth}}}
There's a sharp drop near the end, it's due to us removing support for Jax and TensorFlow library-wide.
Of course, it is not only this effort that allowed to reduce the maintenance load.
A related optimization was the following one. You've likely heard about flash attention and its several variants.
The attention computation itself happens at a lower level of abstraction than the model itself.
However, we were adding specific torch operations for each backend (sdpa, flash-attention iterations, flex attention) but it wasn't a minimal user api.
External Attention classes
We moved to an attention interface that allowed the following:
We keep a Callable for the naive implementation of the attention, called "eager" computation. This Callable is named eager_attention_forward, and can be run as long as the user had torch installed, which is a requirement in any case.
In other words, we moved from a class interface to a function interface: in order to use more complex attention implementations, the config is checked, and can use other Callables, including kernel bindings that are much faster, if they are available.
This exemplifies the fact that we prefer to have an interface that is standard, but not abstract.
attention_interface: Callable = eager_attention_forward
if self.config._attn_implementation != "eager":
attention_interface = ALL_ATTENTION_FUNCTIONS[self.config._attn_implementation]
A strength of the new attention interface is the possibility to enforce specific kwargs, which are needed by kernel providers and other dependencies. We know that kwargs are often a necessary evil that plagues tools with widespread compatibility; and it is something we have aimed to reduce, and will continue reduce in order to improve readability - with them, the current system is a minimal user api.
For better information, we plan to use python features such as Annotated for example, to inform users of what we expect typically in an argument. That way, higher-level information could be included directly in the type annotations, like so (tentative design):
from typing import Annotated
MyModelOutputAnnotated = Annotated[MyModelOutput, "shape: (B, C, H, W)"]
eager_attention_forward; faster backends are opt-in via config. We inform via types/annotations rather than enforce rigid kwargs, preserving integrations. Next: distribution concerns are declared as a plan, not model surgery.
Configurable Tensor Parallelism
If you're not familiar with the different flavours of parallelism, I recommend checking out this blog post first, and of course a full dive into the ultra-scale playbook is always recommended.
The essential part is that, as the documentation states when tensors get too large to fit on a single GPU, they are sliced along a particular dimension and every slice is sent to a different GPU.
Why does it matter?
Because we want to avoid code modifications that are unrelated to the model.
We choose to place the level of abstraction higher than the device placement: a matrix multiplication - a nn.Linear layer - should be always expressed in the same way, regardless of how it is placed.
Hence, we want to touch minimally to the modeling code, and only modify it when architectural changes are involved. For instance, for tensor parallelism, we instead now specify a simple tp_plan.
The alternative would be to modify parent classes specific to their
It is written once in the config and passed to .from_pretrained(). The plan maps module name patterns to partitioning strategies. Strategies are resolved by the internal ParallelInterface, which wires to sharding implementations ColwiseParallel, RowwiseParallel, packed variants, and so on.
{{{fragment-tp-plan}}}
Which allows a user to run with multiple processes per node, e.g. 4 GPUs:
torchrun --nproc-per-node 4 demo.py
Semantics stay in the model (a Linear stays a Linear), distribution is orthogonal and declared via strings: "colwise" splits columns of weights/bias across ranks; "rowwise" splits rows; packed variants shard fused weights; The mapping keys accept glob patterns like layers.*.mlp.down_proj to target repeated submodules.
tp_plan), not edits to Linears. Glob patterns target repeated blocks; modeling semantics stay intact. Next: per-layer attention/caching schedules declared in config, not hardcoded.
Layers, attentions and caches
Following the same logic, the nature of attention and caching per layer of a model should not be hardcoded. We should be able to specify in a configuration-based fashion how each layer is implemented. Thus we defined a mapping that can be then
ALLOWED_LAYER_TYPES = (
"full_attention",
"sliding_attention",
"chunked_attention",
"linear_attention",
...
)
and the configuration can be explicit about which attention type is in which layer, see e.g. gpt-oss, which alternates sliding and full attention:
"layer_types": [
"sliding_attention",
"full_attention",
...,
"sliding_attention",
"full_attention"
],
This is minimal to implement on the user side, and allows to keep the modeling untouched. It is also easy to tweak.
Community Kernels
The same principle extends to normalization, activation, and other code paths. The model defines semantics; a kernel defines how to execute them faster. We annotate the module to borrow a communityโprovided forward, keeping a consistent public surface
@use_kernel_forward_from_hub("RMSNorm")
class GlmRMSNorm(nn.Module):
...
Plus, this opened another angle of contribution for the community. People who are GPU whisperers can now contribute optimized kernels. You can check on the kernel community blog post to learn more about it!
Even more resources have been added, like the formidable kernel builder with its connected resources to help you build kernels with it and with nix.
Modular developments
Now, we have a form of inheritance in our codebase. Some models become standards, and model contributors are given the opportunity to define standards. Pushing the boundaries of scientific knowledge can translate into the boundaries of engineering if this effort is made, and we're striving for it. It's hard to conceptualize very large libraries and how their components interact with each other, regardless of your cognitive abilities for abstractions. So I wanted to take a look at the current state of modularity across the repository. How many models are defined using components of others?
To get this graph, I used the heuristic of modular inheritance.
- Does this model have a
modularfile? - In this
modularfile, what models, configurations and processings are imported? - Recurse through the model list that way.
So what do we see? Llama is a basis for many models, and it shows. Radically different architectures such as mamba have spawned their own dependency subgraph.
{{{fragment-dependency-graph}}}
However, even if llava defines a few VLMs, there's far too many vision-based architectures that are not yet defined as modulars of other existing archs. In other words, there is no strong reference point in terms of software for vision models. As you can see, there is a small DETR island, a little llava pocket, and so on, but it's not comparable to the centrality observed for llama.
Another problem is, this is only for modular models. Several models do NOT have a modular file.
How do we spot them, and how do we identify modularisable models?
Many models, but not enough yet, are alike
So I looked into Jaccard similarity, which we use to measure set differences. I know that code is more than a set of characters stringed together. I also used code embedding models to check out code similarities, and it yielded better results, for the needs of this blog post I will stick to Jaccard index.
It is interesting, for that, to look at when we deployed this modular logic and what was its rippling effect on the library. You can check the larger space to play around, but the gist is: adding modular allowed to connect more and more models to solid reference points. We have a lot of gaps to fill in still.
{{{fragment-model-timeline}}}
If you've checked out llava, you've seen that llava_video is a red node, connected by a red edge to llava: it's a candidate, something that we can likely remodularize, not touching the actual model but being much more readable with DRY*.
llava_video โ llava) for refactors that preserve behavior. Next: concrete VLM choices that avoid leaky abstractions.
VLM improvements, avoiding abstraction
We don't have cookbook for common VLM patterns (image token scatter, multiโtower encoders, crossโattn bridges). This is one of the main improvement points where we can work.
For instance, we thought of abstracting away the mixing of inputs_embeds, the tensor fed into an llm decoder in 95% of the existing VLMs. It would have looked like something like
class InputsEmbeddingMixerMixin(nn.Module):
#
But this is abstracting away an important component of the modeling.. Embedding mixin is part of the model, removing it would break it. A user opening modeling_qwen2.5_vl should not have to go to another file to understand how it works.
This is the current state of abstractions across a modeling file:
The following Pull request to standardize placeholder masking is a good example of what kind of changes are acceptable. In a VLM, we always need to insert embeddings from various encoders at various positions, so we can have a function to do it. For Qwen2 VL, for instance, it will look like this:
def get_placeholder_mask(
self,
input_ids: torch.LongTensor,
inputs_embeds: torch.FloatTensor,
image_features: torch.FloatTensor = None,
video_features: torch.FloatTensor = None,
):
"""
Obtains multimodal placeholdr mask from `input_ids` or `inputs_embeds`, and checks that the placeholder token count is
equal to the length of multimodal features. If the lengths are different, an error is raised.
"""
if input_ids is None:
special_image_mask = inputs_embeds == self.get_input_embeddings()(
torch.tensor(self.config.image_token_id, dtype=torch.long, device=inputs_embeds.device)
)
special_image_mask = special_image_mask.all(-1)
special_video_mask = inputs_embeds == self.get_input_embeddings()(
torch.tensor(self.config.video_token_id, dtype=torch.long, device=inputs_embeds.device)
)
special_video_mask = special_video_mask.all(-1)
else:
special_image_mask = input_ids == self.config.image_token_id
special_video_mask = input_ids == self.config.video_token_id
n_image_tokens = special_image_mask.sum()
special_image_mask = special_image_mask.unsqueeze(-1).expand_as(inputs_embeds).to(inputs_embeds.device)
if image_features is not None and inputs_embeds[special_image_mask].numel() != image_features.numel():
raise ValueError(
f"Image features and image tokens do not match: tokens: {n_image_tokens}, features {image_features.shape[0]}"
)
n_video_tokens = special_video_mask.sum()
special_video_mask = special_video_mask.unsqueeze(-1).expand_as(inputs_embeds).to(inputs_embeds.device)
if video_features is not None and inputs_embeds[special_video_mask].numel() != video_features.numel():
raise ValueError(
f"Videos features and video tokens do not match: tokens: {n_video_tokens}, features {video_features.shape[0]}"
)
return special_image_mask, special_video_mask
But this is within the modeling file, not in the PreTrainedModel base class. It will not move away from it, because it'd break the self-contained logic of the model.
PreTrainedModel. Next: pipeline-level wins that came from PyTorch-first choices (fast processors).
On image processing and processors
Choosing to be a torch-first software meant relieving a tremendous amount of support from jax and TensorFlow , and it also meant that we could be more lenient into the amount of torch-dependent utilities that we were able to add. One of these is the fast processing of images. Where they were before assumed to be minimal ndarrays, making stronger assumptions and enforcing torch and torchvisionnative inputs allowed up to speed up massively the processing time for each model.
The gains in performance are immense, up to 20x speed for most models when compiled torchvision ops. Further, it allows to have the whole pipeline solely on GPU.
Thanks Yoni Gozlan for the great work!
Reduce barrier to entry/contribution
This is an overall objective: there's no transformers without its community.
Having a framework means forcing users into it. It restrains flexibility and creativity, which are the fertile soil for new ideas to grow.
Among the most valuable contributions to transformers is of course the addition of new models. Very recently, OpenAI added GPT-OSS, which prompted the addition of many new features to the library in order to support their model.
A second one is the ability to fine-tune and pipeline these models into many other softwares. Check here on the hub how many finetunes are registered for gpt-oss 120b, despite its size!
Models popularity
Talking about dependencies, we can take a look at the number of downloads for transformer models popularity. One thing we see is the prominence of encoders: This is because the usage of encoders lies in embeddings, just check out EmbeddingGemma for a modern recap. Hence, it is vital to keep the encoders part viable, usable, fine-tune-able.
{{{fragment-model-visualisation}}}
As the codebase grows, with our friend codebase Sentence Transformers, we need to maintain this one as well. Retrieval use-cases, smart dbs, like FAISS-based indexing rely on it, and thus indirectly on transformers.
In that regard, we DO want to be a modular toolbox, being minimal enough and well documented enough so any ML/AI developer can use transformers without having to think about it. We aim to reduce the cognitive load brought about by model development, not increase it.
So, how do these design choices, these "tenets" influence development of models and overall usage of transformers?
A surgical toolbox for model development
Attention visualisation
All models have the same API internally for attention computation, thanks to the externalisation of attention classes. it allows us to build cool tools to visualize the inner workings of the attention mechanism.
One particular piece of machinery is the attention mask. Here you see the famous bidirectional attention pattern for the whole prefix (text + image) in PaliGemma and all Gemma2+ models, contrasting with the usual "causal-only" models.
{{{fragment-attention-visualizer}}}
Logging entire model activations
Further, because it is all PyTorch (and it is even more now that we support only PyTorch), we can easily debug any model when we want to add it to transformers. We now have a power-user tool for porting or adding models, that wraps a forward pass, intercepts every submodule call, and logs shapes, dtypes, and sample statistics of inputs/outputs to nested JSON.
It just works with PyTorch models and is especially useful when aligning outputs with a reference implementation, aligned with our core guideline.
Cooking faster CUDA warmups
Having a clean external API allows us to work on the true inner workings of transformers. One of the few recent additions was the CUDA warmup via caching_allocator_warmup which improved massively the loading footprint by pre-allocating GPU memory to avoid malloc bottlenecks during model loading, achieving a 7x factor for an 8B model, 6x for a 32B, you can check out the source!
{{{fragment-warmup_demo}}}
It's hard to overstate how much of a lifesaver that is when you're trying to load a model as fast as possible, as it's the narrowest bottleneck for your iteration speed.
Transformers-serve and continuous batching
Having all these models readily available allows to use all of them with transformers-serve, and enable interfacing with them with an Open API-like pattern. As a reminder, the hub also opens access to various inference providers if you're interested in model deployment in general.
transformers serve
curl -X POST http://localhost:8000/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{"messages": [{"role": "system", "content": "hello"}], "temperature": 0.9, "max_tokens": 1000, "stream": true, "model": "Qwen/Qwen2.5-0.5B-Instruct"}'
This provides an OpenAI-compatible API with features like continuous batching (also check here) for better GPU utilization.
Continuous batching is in itself very much linked to the great work of vLLM with the paged attention kernel, further justifying the facilitation of external kernels.
Community reusability
Transformers-serve is transformers-first, for sure, but the library is made first and foremost to be reused at large by the open-source ecosystem.
Adding a model to transformers means:
- having it immediately available to the community
- having it immediately usable in vLLM, SGLang, and so on without additional code. In April 2025, transformers was added as a backend to run models on vLLM, which optimizes throughput/latency on top of existing transformers architectures as seen in this great vLLM x HF blog post.
This cements the need even more for a consistent public surface: we are now a backend, and there's more optimized software than us to handle serving. At the time of writing, more effort is done in that direction. We already have compatible configs for VLMs for vLLM (say that three times fast), here for GLM4 video support, and here for MoE support for instance.
What is coming next
The next major version of transformers is just around the corner. When v5 is releasd, backwards compatibility will try to stay as solid as possible. Changes we do now are to ensure this.
Instead, what we aim to be is way more of a modular Toolbox. What we are not is a framework: you should not be FORCED to rewrite every modeling, but it is better for your model to be able to inherit from PreTrainedModel and have enabled TensorParallel, from_pretrained, sharding, push_to_hub, loss, as well as PEFT/TRL/SGLang/vLLM and other fine-tuning and fast inference options.