nebulette/recurrent-nora-1.6b

Recurrent Puro

The text encoder can predict the logits behind the <|mask|> tokens.

Regardless of the captioning strategy, it can understand both booru captions and natural language.

The loss was calculated in the ForMaskedLM mode during training.

The Puro 2B contrastive was used as parent model.

The details of the reused datasets can be found here.

Expansion vs layer pruning

Gazingstar's model doubles each block. Meanwhile, this text encoder runs the same layer twice: once without NoRA and once with it.

The loss calculation then compares the second run to the output of the parent layer.

recoilme has repeated the mid blocks two times with a lightweight embedding in between.

In this work, the DecoderLayer has two (super) forward passes; the second call attempts to approximate the output of the removed layer.

Differences from recurrent Gemma

nightknocker has removed some layers from the encoder, then retrained the whole model.

This work focuses on lost information by replicating each parent layer's behaviour. The loss was not only calculated at the final layer.

This model is different from IRAFM in that it targets all linear modules in the layers through NoRA.

Not as simple as:

x = adapter(torch.cat([x, input_embeds], dim=-1))

Parent-weight initialization

The NoRA weights were precalculated in the initialization stage, based on the difference between the layer that was kept and the subsequent removed layer.

The recurrent model converges more slowly than its parent and produces slightly different output.

References

• 2502.05171
• 2511.07384
• 2601.08584
• 2608.11233
• 2608.31036
• 2609.01343