lambdalabs

/

pythia-1.4b-deduped-synthetic-instruct

like 4

I was trying to run this code:

for param in model.parameters():
    param.requires_grad = False  # freeze the model - train adapters later
    if param.ndim == 1:
        # cast the small parameters (e.g. layernorm) to fp32 for stability
        param.data = param.data.to(torch.float32)

model.gradient_checkpointing_enable()  # reduce number of stored activations
model.enable_input_require_grads()

class CastOutputToFloat(nn.Sequential):
    def forward(self, x): return super().forward(x).to(torch.float32)

model.lm_head = CastOutputToFloat(model.lm_head)

And got this error:

AttributeError: 'GPTNeoXForCausalLM' object has no attribute 'lm_head'

Please help.