A modified DNABERT2 that returns the attention too
#24
by
jaandoui
- opened
I have modified the code on the original repository to extract attention too. Since I changed many parts, I uploaded them into a new public repository. Zhihan you can analyze and use it for your model if you yourself are interested in the attention. The name of HuggingFace repo is jaandoui/DNABERT2-AttentionExtracted and here is a link https://huggingface.co/jaandoui/DNABERT2-AttentionExtracted/blob/main/bert_layers.py
Feel free to send me some feedback as my implementation is not perfect (but works correctly for me). The code is simply returning attention tensor instead None when output_attentions is enabled.
Read the first part of README and Bert_Layers.py file to understand the changes.