Results & Discussion 1 Simple embedding-based transfer almost as good as CAFA3 top-10. First, we predicted GO 2 terms for all 3,328 CAFA3 targets using the Gene Ontology Annotation (GOA) data set GOA2017 3 (Methods), removed all entries identical to CAFA3 targets (PIDE=100%; set: GOA2017-100) and 4 transferred the annotations of the closest hit (k=1; closest by Euclidean distance) in this set to 5 the query. When applying the NK evaluation mode (no-knowledge available for query, 6 Methods/CAFA3), the embedding-transfer reached Fmax scores (Eqn. 3) of 37±2% for BPO 7 (precision: P=39±2%, recall: R=36±2%, Eqns. 2/3), F1=50±3% for MFO (P=54±3%, R=47±3%), 8 and F1=57±2% for CCO (P=61±3%, R=54±3%; Table 1 , Fig. 1, Fig. S1 ). Errors were estimated 9 through the 95% confidence intervals (± 1.96 stderr). Replacing the Euclidean by cosine distance 10 (more standard amongst those working with embeddings, e.g. in NLP) changed little (Table 1;   11 for simplicity, we only used Euclidean from here on). In the sense that the database with 12 annotations to transfer (GOA2017) had been available before the CAFA3 submission deadline closest database protein (dark bar) for the three ontologies (BPO, MFO, CCO) to the top ten methods that 25 -in contrast to our method -did compete at CAFA3 and to two background approaches "BLAST" 26 (homology-based inference) and "Naïve" (assignment of terms based on term frequency) (lighter bars). 27 The result shown holds for the NK evaluation mode (no knowledge), i.e. only using proteins that were novel 28 in the sense that they had no prior annotations. If we had developed our method before CAFA3, it would 29 have almost reached the tenth place for MFO and CCO and ranked even slightly better for BPO. Error 30 bars (for our method) marked the 95% confidence intervals. GOA2017X (2017), and GOA2020-100 (2020) used for annotation transfer (note: the notation '-5 100' implies that any entry in the data set with PIDE=100% to any CAFA3 protein had been 6 removed). All values were compiled for picking the single top hit (k=1) and using the CAFA3 targets 7 from the NK and full evaluation mode 22 . For all simple annotation transfers (embedding-and 8 sequence-based), performance was higher for the more recent data sets (GOA2020 vs. BPO than expected from this distribution even for k=1. Only for MFO the average (11.7 terms) 20 predicted was slightly lower than expected for k=1 (number of terms exploded for k>1: Table S3 ). 21 While precision dropped with adding terms, recall increased (Table S3) annotations (LK evaluation mode) than for novel proteins without any annotations (NK evaluation 30 mode; Table 1) ; the same was true for the CAFA3 top-10 for which the Fmax scores increased 31 even more than for our method for BPO and MFO, and less for CCO (Fig. 1, Fig. S2 Fig. 1 ). This suggested embeddings 6 to capture information beyond raw sequences. Explicitly calculating the correlation between 7 sequence and embedding similarity for 431,224 sequence pairs from CAFA3/GOA2017-100, we 8 observed a correlation of r=0.29 (Spearman's correlation coefficient, p-value < 2.2e-16; Table   9 2). Thus, sequence and embedding similarity correlated at an unexpectedly low level. However, 10 our results demonstrated that embedding similarity identified more distant relatives than 11 sequence similarity (Figs. S1, S4). were within the 95% confidence interval (Fig. 2) . This clearly showed that our approach benefited 36 from information available through embeddings but not through sequences, and that at least completely independent of the sequence similarity between query and template (protein of known function). 16 The observation that all bars tended to fall toward the right implied that embedding and sequence similarity 17 correlated (although for CCO, Fmax remained within the 95% confidence interval of Fmax for GOA2017-100). 18 The observation that our method remained mostly above the baseline predictions demonstrates that Many of the non-experimental annotations constituted sequence-based annotation 5 transfers. Thus, non-experimental annotations might have helped because they constituted an 6 implicit merger of sequence and embedding transfer. Adding homologous proteins might "bridge" 7 sequence and embedding space by populating embedding space using annotations transferred 8 from sequence space. The weak correlation between both spaces supported this speculation 9 because protein pairs with very similar sequences may differ in their embeddings and vice versa. (Table 1) . This constituted a substantial increase over GOA2017-100 (Table 1) . 19 The large performance boost between GOA2017 and GOA2020 suggested the addition 20 of many relevant GO annotations. However, for increasingly diverged pairs (Q,T), we observed 21 a much larger drop in Fmax than for GOA2017 (Fig. 2, Fig. S3 ). In the extreme, GOA2020-20 22 (PIDE(Q,T)<20%) with Fmax=33±2% (BPO), 44±2% (MFO), and 54±2% (CCO) fell to the same 23 level as GOA2017-20 (Fig. 2, Fig. S3 ). These results suggested that many of the relevant GO 24 annotations were added for proteins sequence-similar to those with existing annotations. Put 25 differently, many helpful new experiments simply refined previous computational predictions. 26 Running BLAST against GOA2020-100 for sequence-based transfer (choosing the hit 27 with the highest PIDE) showed that sequence-transfer also profited from improved annotations 28 (difference in Fmax values for BLAST in Table 1 precision increased, and recall decreased as expected (Fig. 3) . While precision increased up to 14 82% for BPO, 91% for MFO, and 70% for CCO, it also fluctuated for high RIs (Fig. 3) . This trend 15 was probably caused by the low number of terms predicted at these RIs. For CCO, the RI 16 essentially did not correlate with precision. This might point to a problem in assessing annotations 17 for which the trivial Naïve method reached values of Fmax~55% outperforming most methods. 18 Possibly, some prediction of the type "organelle" is all that is needed to achieve a high Fmax in 19 this ontology. better predictions than relatively high embedding RIs (Fig. S4) . Overall, these results (Fig. S4 ) 25 again underlined that LM embeddings abstract information from sequence that are relevant for 26 comparisons and not captured by sequences alone. However, it also indicates that even protein 27 pairs with low embedding similarity can share similar GO terms. In fact, embedding similarity only 28 weakly correlated with GO term similarity (Spearman rank coefficient (r=0.28, p-value < 2.2e-29 16), but proteins with identical GO annotations were on average more likely to be close than 30 proteins with more different GO annotations (Fig. S5) . The similarity of GO terms for two proteins 31 was proxied through the Jaccard index (Eqn. 7). More details are provided in the SOM. 32 To benefit from the cases where BLAST outperformed our approach, we tried simple 33 combinations: firstly, we considered all terms predicted by embeddings from either SeqVec or 34 ProtBert. Secondly, reliability scores were combined leading to higher reliability for terms 35 predicted in both approaches than for terms only predicted by one. None of those two improved 36 performance (Table S4, (Table S4 summarized all predicted and   25 annotated GO leaf terms, the corresponding names can be found in the additional files 26 predictions_$emb_$ont.txt).
Step 2 (Fmax(embedding)-Fmax(sequence)), MFO +8±3%, and CCO +11±2% (Table 1, Fig. 1) ; it even 12 might have reached the top ten, had it participated at CAFA3 (Fig. 1) . Embedding-based transfer 13 remained above the average for sequence-based transfer even for protein pairs with PIDE<20% 14 (Fig. 2) , i.e. embedding similarity worked for proteins that diverged beyond the recognition in embedding-based annotation transfer significantly (Table 1) . Another aspect of the simplicity is 20 that, at least in the context of the CAFA3 evaluation, the choice of none of the two free parameters 21 really mattered: embeddings from both LMs tested performed, on average, equally, and the 22 number of best hits (k-nearest neighbors) did not matter much (Table S2 ). The power of this new 23 concept is generated by the degree to which embeddings implicitly capture important information 24 relevant for protein structure and function prediction. Ultimately, the reason for this was in that 25 the correlation between embeddings and sequence remained limited ( Table 2) .
and each direction a 512-dimensional vector; in this work, only the forward and backward passes 48 of the first LSTM layer were extracted and concatenated into a matrix of size L * 1024 for a protein with L residues. A fixed-size representation was then derived by averaging over the length 1 dimension, resulting in a vector of size 1024 for each protein (Fig. S7 ). This simple way of 2 information pooling outperformed in many cases more sophisticated methods in NLP 49 and 3 showed competitive performance in bioinformatics for some tasks 23,24,29 .
To evaluate the effect of using different LMs to generate the embeddings, we also used 5 a transformer-based LM trained on protein sequences (ProtBert-BFD 24 , here simply referred to 6 as ProtBert). ProtBert is based on the LM BERT 40 (Bidirectional Encoder Representations from 7 Transformers 50 ) which processes sequential data through the self-attention mechanism 51 . Self-8 attention compares all tokens in a sequence to all others in parallel, thereby capturing long-range 9 dependencies better than LSTMs. BERT also replaced ELMo's auto-regressive objective by 