Spaces:

AI4PD

/

hexviz

like 6

Sleeping

We're applying for a GPU grant for hexviz. We hope it can allow people to analyze attention patterns on protein structures, enabling one to peek inside protein language models and perhaps learn something about the language of proteins.

The hope for the tool is captured in this quote from "BERTology meets biology" (https://arxiv.org/abs/2006.15222)

In order for learned representations to be accessible to domain experts, they must be presented in an appropriate context to facilitate discovery. Visualizing attention in the context of protein structure is one attempt to do so.

There are great tools for attention analysis in the context of NLP like https://exbert.net/, but none for protein structure and protein language models that we know of.

With the grant, we would be able to keep bigger and closer to state-of-the-art models in memory for inference.
We really want to be able to do the attention analysis for the popular protein models like https://huggingface.co/Rostlab/prot_t5_xl_half_uniref50-enc and/or https://huggingface.co/nferruz/ProtGPT2 as we think analyzing them would be valuable and interesting.

The tool is still in development. We expect interest from the growing AI for proteins community once we announce in not too long.
The GPU grant would be valuable now to be able to develop and adjust the tool for the larger models we (and others) are most interested in studying (we're limited by memory to only small models now).

Thanks for considering the application!
I'm from the AI for protein design lab in Barcelona https://www.aiproteindesign.com/ 👋

Interactive attention on protein structure view	(WIP) view to identify heads of interest in the model