Discrepancy in Parameter Count: A Closer Look at the Model's Size and the Number of Layers
#3
by
Karim-Gamal
- opened
How are you claiming that your model has only 21M parameters when I verified it and found that it actually has over 117M parameters and the Number of Layers: 199 not 12?
from transformers import AutoModel
model_name = "microsoft/Multilingual-MiniLM-L12-H384" # Total Trainable Parameters: 117653760 Not 21M
model = AutoModel.from_pretrained(model_name)
n_layers = len([f for f in model.parameters()])
n_params = sum(p.numel() for p in model.parameters() if p.requires_grad)
print("Number of Layers:", n_layers)
print("Total Trainable Parameters:", n_params)
# Number of Layers: 199 not 12
# Total Trainable Parameters: 117653760 not 21M
Karim-Gamal
changed discussion title from
Discrepancy in Parameter Count: A Closer Look at the Model's Size
to Discrepancy in Parameter Count: A Closer Look at the Model's Size and the Number of Layers
How are you claiming that your model has only 21M parameters when I verified it and found that it actually has over 117M parameters and the Number of Layers: 199 not 12?
from transformers import AutoModel model_name = "microsoft/Multilingual-MiniLM-L12-H384" # Total Trainable Parameters: 117653760 Not 21M model = AutoModel.from_pretrained(model_name) n_layers = len([f for f in model.parameters()]) n_params = sum(p.numel() for p in model.parameters() if p.requires_grad) print("Number of Layers:", n_layers) print("Total Trainable Parameters:", n_params) # Number of Layers: 199 not 12 # Total Trainable Parameters: 117653760 not 21M
@Karim-Gamal The Transformer encoder's parameter count is 21M, without the word embeddings.
unilm
changed discussion status to
closed
I got your point, thanks!