feat: choose flash attention heuristically if not set explicitly

fix: assert is None for other kwargs too

feat: added head_mask

fix: cast mask to bool

fix: move flash components into top-level

feat: try to fix import error

feat: moved flash attention code into this repository

feat: added encode method

fix: try to skip initialization of task type embeddings

fix: try to skip initialization of task type embeddings

feat: added option for QK normalization

feat: implement task type embeddings (#1)

feat: added back option not to use flash attention

feat: support gradient checkpointing

removed unused imports

removed __init__ from BertPretrainedModel

added config_class and base_model_prefix

Fixed typo

Fixed typo

Try to subclass PretrainedModel

Try to subclass PretrainedModel

strict=True for debugging

try to simplify checkpointing

removed debugging

debugging

debugging

debugging assertion

fix: fixed get_input_embeddings method

feat: added get_input_embeddings method to BertForPreTraining

feat: fixed _from_config

removed from_config

fix: try to get from_config to work

feat: added from_config, also pass additional kwargs from config to model

feat: updated modeling_bert.py to allow MLM-only training

feat: reverted monkey patch

feat: try to monkey-patch index_first_axis

feat: try to fix compilation

feat: added debug print

feat: updated .to() override to handle kwargs

Revert "feat updated debug pring"

feat updated debug pring

fix: try to fix .to(torch.float16) with ALiBi

Revert "feat: added back option to disable flash attention"

feat: added back option to disable flash attention

fix: always use flash attention

revert

fix: fix typo

fix: try to fix AutoModel.from_config

fix: try to fix AutoModel.from_pretrained

fix: try to fix import error