Paper: Generative modeling for RNA splicing code predictions and design (ICLR-2024)

Reviewer comment (Evaluation):
Traspr appears to excel at predicting constitutive and non-constitutive exons but performs poorly for exons that fall in between (the truly alternatively spliced exons). It might provide more insight if the tested exons were categorized accordingly (e.g., see Fig. 1C in the SpliceAI paper).

Author response:
>Traspr appears to excel at predicting constitutive and non-constitutive exons but performs poorly for exons that fall in between (the truly alternatively spliced exons). It might provide more insight if the tested exons were categorized accordingly (e.g., see Fig. 1C in the SpliceAI paper).

The above comment seems to reflect a fundamental misunderstanding of the analysis/data which is a clear indication we failed to explain it well. Alternatively, there is a lack of clarity regarding the terms “constitutive” and “non-constitutive” used above by the Reviewer. Either way, let us try to clarify this important point. First,  ALL the data presented in this work is for alternative cassette exons. Alternative exons are defined as those for which we have evidence of being alternative based on the annotation and/or read data as processed by the MAJIQ algorithm for the specified tissues. Conversely, “Constitutive” are those that appear as such in the annotation and don’t have read data supporting otherwise. Notably, these labels are by definition somewhat noisy and context specific: A constitutive exon may become alternative in some other context and an alternative one may have very little reads (or just annotation) supporting it as such while in practice it is very much “constitutive”. Distinguishing between alternative and constitutive exons (originally based on annotation alone) is considered a relatively easy task in the splicing field with AUC well over 90% in SVM papers dating back to 2005 (Dror et al Bioinformatics) and later in Splicing code papers as well (~97% AUC, Barash et al Genome Biology 2014). This is NOT the task we are handling here as all exons are “alternative” by the above definition. In fact, if we were to introduce such constitutive exons into our test set our performance would skyrocket (as discussed in the GB 2014 paper above). This is also not about scoring (cryptic/weak) splice sites vs ‘main’ ones and mutations that activate/decrease those (as in the SpliceAI paper). Consequently, referring to Fig 1C in the SpliceAI the reviewer pointed us to, we see a significant difference: The vast majority of inclusion rates in that figure (>85%) are 0.9-1.0. In contrast, the data in the GTEX-3 set we described above, based on alternative exons in those tissues, is distributed more evenly with ~50%  at the 0.9-1.0 range and the rest are roughly evenly distributed between the 0-0.1 and the 0.1-0.9 ranges. 

Another option is that the Reviewer made here a general comment that high inclusion and high exclusion are generally easier to predict. We would be the first to agree (see above stats from scoring constitutive vs. alternative exons) and there are of course biological reasons for that - easier to detect a crappy splice site or a very strong (aka constitutive) exon. Furthermore, “in between” cases as the Reviewer calls them are more likely to be those that are shifting in a tissue specific manner - these are exactly the ones we are trying to better predict here.

In conclusion, predicting alternative vs. constitutive is a much easier task then predicting exact PSI for alternative only events (which is the case here), and predicting ‘in between’ cases specifically is harder as these are harder to detect and many times reflect tissue specific signals. However, we strongly disagree with the negative conclusion “performs poorly” - compared to what? For sure we could do batter, but we believe we show strong evidence we actually perform better than previous algorithms on those in between values (see for example the GTEX-3C evaluations above when things by definition shift from 0 and 1 and classifying tissue specific changes).

Reviewer comment (Experiments):
The authors only examine Daam1 mutations published in (Barash et al., 2010), while numerous datasets have been published since then, e.g., in (Xiong et al. 2015) and (Jaganathan et al., 2019), among many others.

Author response:
>The authors only examine Daam1 mutations published in (Barash et al., 2010), while numerous datasets have been published since then, e.g., in (Xiong et al. 2015) and (Jaganathan et al., 2019), among many others.

The reviewer brings up an important point which seems that we failed to communicate clearly. Let us try to amend this here (and, if given the opportunity, in the revised version). The tasks in this paper were to (a) predict tissue specific splicing changes (b) build generative models to generate sequences with tissue specific splicing profiles. We stress the task was NOT to assess mutations - we emphasized this as a direction of future work in the discussion, well outside the scope of this work. The only reason to include the Daam1 results, and the ENCODE RBP analysis, was to show the model is capturing tissue specific regulatory signals. The mutations in Xiong et al 2015 and Jaganathan et al 2019 are NOT, to the best of our knowledge, for tissue specific splicing regulation and therefore we did not see a specific reason to include these (and similar datasets) in our results.

With that said, the combination of high confidence (level 5) and skeptical tone have led us to believe that if we don’t do additional mutation analysis the reviewer will again suspect our work. Thus, we have worked hard to include a much more comprehensive mutagenesis dataset. We decided to focus our effort on a recent dataset from Cortés-López et al Nat Comm 2022. We choose this dataset because: (1) It focuses on splicing of CD19 exon 2 which has significant implications for immunotherapy treatment (2) Because of that, the authors included exon 1-2-3 in their mini gene construct, including the flanking introns (3) consequently it has >10K data points of random mutations (SNP/indels) combinations across both the exon and the flanking intron (no specific selection bias), > 4K mutation positions with 90% of the positions covered by at least 4 different mutation combinations (4) unlike some other studies the authors quantified PSI directly (and not some related measure) and the replicates had > 0.9 correlations. We call this dataset the CD19 data for short.

After some data cleaning/filtering for consistent measurements etc. we first trained TrASPr on the cell line used in this study (NLM6) and then fine tuned it to predict the CD19 data. We evaluated TrASPr on this dataset using 3 different test set definitions:

*RCV*: Random cross validation. This means the model may have seen a specific mutation, but not the specific combination of mutations in the test set.

*SPF*: Single position filtering. Here the test data involves mutation which we have never seen before in any training case. Due to this constraint the data is more limited: Training set: 4628, Test set:1480

*SPFWN*: Single position filtering with neighbors. Here we do not only hide the test positions, but also any position that ended up in a mutation combination with those positions. The nature of the dataset (mutation combination) is such that even when we do this for positions with least amount of shared mutations we still end up with Training set: 2300, Test set: 101.

TrASPr results for these tests (PSI Pearson correlation) are:

RCV: 0.93

SPF: 0.91

SPFWN: 0.85 

Using the same test sets on Pangolin and SpliceAI we get:

**Splice AI:**

SPF 0.7

SPFWN: 0.75

**Pangolin:**

SPF 0.58

SPFWN: 0.44

Reviewer comment (Experiments):
When evaluating the Bayesian optimization (BO) method for sequence design, other datasets could be considered, such as the SMA dataset in (Xiong et al. 2015).

Author response:
>When evaluating the Bayesian optimization (BO) method for sequence design, other datasets could be considered, such as the SMA dataset in (Xiong et al. 2015).

Again, that data is not tissue specific and our main analysis is about constraining BOS to tissue specific signals. We agree additional assessment of that sort for BOS would be nice but we simply were not able to find a good/easy to process dataset for that and definitely not able to complete any additional analysis with the week long of rebuttal time.

Reviewer comment (Experiments):
Is the Transformer model truly necessary here? Could the VAE be used to obtain representations and then be fed into the MLP? Essentially, could a single splice junction model be trained instead of a Transformer and a VAE?

Author response:
>Is the Transformer model truly necessary here? Could the VAE be used to obtain representations and then be fed into the MLP? Essentially, could a single splice junction model be trained instead of a Transformer and a VAE?

We have not explored many alternative designs for BOS, we agree it would be interesting to explore alternative design choices in the future.

Reviewer comment (Experiments):
On the same topic, it would be beneficial to understand how a model like SpliceAI, followed by an MLP, would perform as the oracle. I'm not convinced that the Transformer-based splice site detection (with a 400bp context) outperforms SpliceAI with a 10kb context.

Author response:
>On the same topic, it would be beneficial to understand how a model like SpliceAI, followed by an MLP, would perform as the oracle. I'm not convinced that the Transformer-based splice site detection (with a 400bp context) outperforms SpliceAI with a 10kb context.

We hope our detailed analyses described above points to the benefit of using TrASPr and not SpliceAI for designing sequences with a tissue specific splicing profile. The last comment here (“I'm not convinced that the Transformer-based splice site detection with a 400bp context outperforms SpliceAI with a 10kb context.”) is worth discussing though: The reviewer seems to mix here sheer attention region/model size with model choices for different tasks. This is important because this, again, may have led the reviewer to doubt our work. Let us revisit this point: SpliceAI was designed for a very different task - detecting splice sites as these relate to mutations. It has no prior knowledge about where those splice sites are and its approach is “splicing agnostic”: it scans large genomic windows of 10kb long trying to figure out if the middle position will be a splice site yes/no. This is an important and hard task for sure (e.g. disease diagnostics), and SpliceAI excels at it. TrASPr is designed for a very different task - predicing tissue specific splicing changes. Tissue specific splicing changes are hard to predict (see discussion and results above and in the paper) but TrASPr already “knows” where the splice sites are (based on annotation and/or RNASeq information) and we know from decades of research that most of the regulation occurs around those sites. Thus it can “afford” to have only 400bp windows of attention around those splice sites, and yet outperform SpliceAI in this task. Furthermore, as we write in the paper, 10kb is not even enough to cover many of the regions that involve cassette exons, which puts SpliceAI/Pangolin at a disadvantage for this task.

Reviewer comment (Experiments):
Assuming that a tissue-specific splicing model works (which needs to be demonstrated), can the BO method be adapted to modify AS only in a specific tissue, such as the brain?

Author response:
>Assuming that a tissue-specific splicing model works (which needs to be demonstrated), can the BO method be adapted to modify AS only in a specific tissue, such as the brain?

Yes! This is exactly the kind of constraint on BOS that we performed in the paper. Revising the original submission we saw the only description we included was “generating candidate sequences from a given input sequence and a user defined PSI constraint”. The description of that user constraint (for tissue specificity) was omitted. We will fix this in the revised version, given the opportunity.

Reviewer comment (Writing):
Although overall the manuscript is clear and easy to read, it contains many typos and grammatical errors - see below.

Author response:
We thank the reviewer for taking the time to point those out. We will be happy to correct all of those in a revised version, if given the opportunity.

Reviewer comment (Experiments):
When it comes to in-silico mutations, I am not sure the authors' experiment shows all that much. A better approach might be to see how well the model predicts splicing QTLs as was done in the Pangolin paper (and using integrated gradients works better than mutagenesis in our experience).

Author response:
The reviewer raises a good point here we should elaborate on. To be clear, this work aims to achieve two objectives: (a) build a model that achieves SOTA performance for tissue specific splicing predictions (b) develop a new BO method for designing RNA sequence with user defined condition specific splicing profile (and formulating the task as such). We are NOT aiming here to prove SOTA in predicting the effect of mutations for say disease diagnostics. We believe such applications are exciting future directions, as we describe in the discussion section. Thus, the in-silico mutation analysis in this work is only aimed to show the model is able to capture tissue specific regulatory elements. Yes, we very much agree overlapping the model predictions with sQTL is a great way to show the ability to predict the effect of mutations but again this is not the focus of this work and many sQTL hits are not tissue specific. We did not have an easy solution/set to assess against (for example, the set of variants used in the recent Wagner et al Nat Gen 2023 is not accessible to the best of our knowledge) and running an extensive analysis for say sQTL that are also tissue specific is beyond the scope of a short ML conference paper. Please also see a much more lengthy discussion of this point, including additional data and analysis we added for another mutation dataset in response to reviewer Bzy6.

Reviewer comment (Writing):
It is unclear what meta data information could be utilised to predict condition-specific splicing, assuming the input RNA sequences are from the same species and thus are very similar; a concrete example to motivate the proposal would be useful.

Author response:
>Unfortunately, I am not convinced by the manuscript that the goal is achieved. First, I am not sure what meta data information could be utilized to predict condition-specific splicing, assuming the input RNA sequences are from the same species thus are very similar. I feel it will be useful to give a concrete example to motivate the proposal.

There seems to be some lack of clarity regarding what is “meta data”. As we describe in the text, this data is simply information regarding things like exon length, intron length, tissue type etc. This information is known to relate to splicing outcome (e.g. micro exons that are neuro specific) and are thus useful. We clearly show are useful in the ablation study (Table2). It’s true that in some species splicing may be very different (e.g. in plants) but we are not exploring those cases here.

Reviewer comment (Experiments):
The prediction part should be expanded with more experiments to validate the basic assumptions. For example, if the input sequences are replaced with random sequences or sequences from non-splicing genes but with the same meta-data, psi should be close to 0.

Author response:
>Second, I think it will be nice to expand the prediction part with more experiments to validate the basic assumptions. For example, if we replace the input sequences with random sequences or sequences from non-splicing genes, but with the same meta-data, we expect to see that psi is close to 0. Assuming the network has successfully captured the regulatory mechanism, we expect to see the psi will decrease to 0 if we remove the U2 elements (suppl. Fig. 7d) from the input sequences. Since the splicing mechanism is conservative between human and mouse, we expect to see that NN trained on human data should exhibit similar predictive power on mouse homolog genes. If the prediction part works with high precision, then it is worth designing novel RNA sequences. 

We have worked very hard to add analysis and results that would support our claims that (a) we achieve SOTA performance in tissue specific splicing prediction. We refer the reviewer to the extensive details we included in the response to reviewer Bzy6. Regarding the specific suggestions: Abolition of the splice signal (e.g. U2 binding area) leads to immediate destruction of the splicing - this is part of the pre-training of the Transformer on splice sites which is an easy task shown in many previous papers (and why we didn’t emphasize it). More generally, see our discussion in response to Bzy6 regarding the much easier task of constitutive vs. alternative exons.  We do indeed see similar predictive power between mouse and human as we show in the paper (and see again Bzy6 response).

Reviewer comment (Experiments):
If the network has captured the regulatory mechanism, removing the U2 elements (suppl. Fig. 7d) from the input sequences should cause the psi to decrease to 0.

Author response:
>Second, I think it will be nice to expand the prediction part with more experiments to validate the basic assumptions. For example, if we replace the input sequences with random sequences or sequences from non-splicing genes, but with the same meta-data, we expect to see that psi is close to 0. Assuming the network has successfully captured the regulatory mechanism, we expect to see the psi will decrease to 0 if we remove the U2 elements (suppl. Fig. 7d) from the input sequences. Since the splicing mechanism is conservative between human and mouse, we expect to see that NN trained on human data should exhibit similar predictive power on mouse homolog genes. If the prediction part works with high precision, then it is worth designing novel RNA sequences. 

We have worked very hard to add analysis and results that would support our claims that (a) we achieve SOTA performance in tissue specific splicing prediction. We refer the reviewer to the extensive details we included in the response to reviewer Bzy6. Regarding the specific suggestions: Abolition of the splice signal (e.g. U2 binding area) leads to immediate destruction of the splicing - this is part of the pre-training of the Transformer on splice sites which is an easy task shown in many previous papers (and why we didn’t emphasize it). More generally, see our discussion in response to Bzy6 regarding the much easier task of constitutive vs. alternative exons.  We do indeed see similar predictive power between mouse and human as we show in the paper (and see again Bzy6 response).

Reviewer comment (Experiments):
Since the splicing mechanism is conservative between human and mouse, a NN trained on human data should exhibit similar predictive power on mouse homolog genes.

Author response:
>Second, I think it will be nice to expand the prediction part with more experiments to validate the basic assumptions. For example, if we replace the input sequences with random sequences or sequences from non-splicing genes, but with the same meta-data, we expect to see that psi is close to 0. Assuming the network has successfully captured the regulatory mechanism, we expect to see the psi will decrease to 0 if we remove the U2 elements (suppl. Fig. 7d) from the input sequences. Since the splicing mechanism is conservative between human and mouse, we expect to see that NN trained on human data should exhibit similar predictive power on mouse homolog genes. If the prediction part works with high precision, then it is worth designing novel RNA sequences. 

We have worked very hard to add analysis and results that would support our claims that (a) we achieve SOTA performance in tissue specific splicing prediction. We refer the reviewer to the extensive details we included in the response to reviewer Bzy6. Regarding the specific suggestions: Abolition of the splice signal (e.g. U2 binding area) leads to immediate destruction of the splicing - this is part of the pre-training of the Transformer on splice sites which is an easy task shown in many previous papers (and why we didn’t emphasize it). More generally, see our discussion in response to Bzy6 regarding the much easier task of constitutive vs. alternative exons.  We do indeed see similar predictive power between mouse and human as we show in the paper (and see again Bzy6 response).