Protein Ubiquitylation and Sumoylation Site Prediction Based on Ensemble and Transfer Learning Ubiquitylation, a typical posttranslational modification (PTM), plays an important role in signal transduction, apoptosis and cell proliferation. A ubiquitylation like PTM, sumoylation also may affect gene mapping, expression and genomic replication. Over the past two decades, machine learning has been widely employed in protein ubiquitylation and sumoylation site prediction tools. These existing tools require feature engineering, but failed to provide general interpretable features and probably underutilized the growing amount of data. This prompted us to propose a deep learningbased model that integrates multiple convolution and fullyconnected layers of seven supervised learning submodels to extract deep representations from protein sequences and physicochemical properties (PCPs). Especially, we divided PCPs into 6 clusters and customized deep networks accordingly for handling the high correlations among one cluster. A stacking ensemble strategy was applied to combine these deep representations to make prediction. Furthermore, with the advantage of transfer learning, our deep learning model can work well on protein sumoylation site prediction as well after finetuning. On the highquality annotated database SwissProt, our model outperformed several wellknown ubiquitylation and sumoylation site prediction tools. Our code is freely available at https:github.comruiwcodingDeepUbiSumoPre.

Evaluating Protein Transfer Learning with TAPE Abstract  Protein modeling is an increasingly popular area of machine learning research. Semisupervised learning has emerged as an important paradigm in protein modeling due to the high cost of acquiring supervised protein labels, but the current literature is fragmented when it comes to datasets and standardized evaluation techniques. To facilitate progress in this field, we introduce the Tasks Assessing Protein Embeddings (TAPE), a set of five biologically relevant semisupervised learning tasks spread across different domains of protein biology. We curate tasks into specific training, validation, and test splits to ensure that each task tests biologically relevant generalization that transfers to reallife scenarios. We benchmark a range of approaches to semisupervised protein representation learning, which span recent work as well as canonical sequence learning techniques. We find that selfsupervised pretraining is helpful for almost all models on all tasks, more than doubling performance in some cases. Despite this increase, in several cases features learned by selfsupervised pretraining still lag behind features extracted by stateoftheart nonneural techniques. This gap in performance suggests a huge opportunity for innovative architecture design and improved modeling paradigms that better capture the signal in biological sequences. TAPE will help the machine learning community focus effort on scientifically relevant problems. Toward this end, all data and code used to run these experiments are available at https:github.comsonglabcaltape.