M ULTI - SCALE S INUSOIDAL E MBEDDINGS E NABLE L EARNING ON H IGH R ESOLUTION M ASS S PECTROMETRY D ATA
Anonymous authors
Paper under double-blind review
A BSTRACT
Small molecules in biological samples are studied to provide information about disease states, environmental toxins, natural product drug discovery and many other applications.
The primary window into the composition of small molecule mixtures is tandem mass spectrometry (MS2), which produces data that are of high sensitivity and part per million resolution.
We adopt multi-scale sinusoidal embeddings of the mass data in MS2 designed to meet the challenge of learning from the full resolution of MS2 data.
Using these embeddings, we provide a new state of the art model for spectral library search, the standard task for initial evaluation of MS2 data.
We also investigate the task of chemical property prediction from MS2 data, that has natural applications in high-throughput MS2 experiments and show that an average R 2 of 80% for novel compounds can be achieved across 10 chemical properties prioritized by medicinal chemists.
We use dimensionality reduction techniques and experiments with different floating point resolutions to show the essential role multi-scale sinusoidal embeddings play in learning from
MS2 data.
I NTRODUCTION
Tandem Mass Spectrometry (MS/MS) is one of the primary tools used in metabolomics, the characterization of the small molecule ( (cid:62) 1 , 000 Daltons ) contents of complex biological samples.
Tandem
Mass Spectrometry works by measuring with very high precision the masses of molecules and their constituent fragments.
While MS techniques are highly sensitive and extremely precise, inferring the identity of the molecules and their properties from the resulting mass spectra is one of metabolomics’ primary bottlenecks (Dunn et al., 2013).
Unlocking the information contained in mass spectra will lead to an improvement in a number of applications including characterization of disease pathways, development of new agrochemicals, improved forensics analysis, and the discovery of new drugs (Zhang et al., 2020).
Profiling unknown molecules with mass spectrometry consists of several steps.
First, molecules of interest are ionized and separated by their mass to charge ratio ( m/z ), resulting in the MS1 spectrum.
Then, individual “precursor” ions are fragmented, and the m/z ’s of the fragments are recorded in the same manner.
The resulting spectrum contains the m/z ’s and intensities (together, the “peaks”) of all resulting fragments, and is called the MS2 spectrum.
See Glish & Vachet (2003).
In recent years, several machine learning methods have been developed to identify the structures and properties of small molecules from their mass spectra.
These approaches (Huber et al., 2021a;b;
Kutuzova et al., 2021; Litsa et al., 2021; Shrivastava et al., 2021; van Der Hooft et al., 2016) historically discretize m/z (via tokenization or binning).
However, the m/z values obtained in modern mass spectrometry experiments are collected with parts per million levels of precision.
There are three critical reasons to expect that modeling m/z values as numeric quantities is the appropriate technique.
First, we know that relevant chemical information is present at the millidalton level (Jones et al., 2004; Pourshahian & Limbach, 2008) and discretization schemes typically strip this information away.
Second, mass differences between peaks represent fragmentation patterns, and are therefore relevant to understanding the molecule.
Learning to utilize this information from tokens is exceedingly difficult.
Finally, any laboratory measurement always comes with some degree of experimental error, and discretization techniques are particularly susceptible to measurement errors.
In light of this, we proceed by modeling m/z as a numerical value using sinusoidal embeddings, which we hypothesize will enable us to capture information across many orders of magnitude.
In this work, we apply a numerical representation of m/z that uses sinusoidal embeddings across multiple scales to retain the information content of MS2 data across its entire mass resolution.
We use these embeddings to achieve state of the art accuracy on spectral library search (Stein &
Scott, 1994), a task which identifies compounds by comparing mass spectra from experiments to spectra from previously characterized compounds.
This task is the primary method used in standard metabolomics analyses, but is challenging because spectra from a given compound vary widely according to experimental conditions.
We also investigate the task of predicting chemical properties relevant to drug discovery from MS2 data and apply the same modeling approach to this task.
We achieve 80% average R 2 for out of sample molecules across 10 properties on this task, which is high enough to enable first-pass filtering and selection of candidate drug molecules in high-throughput experiments based solely on spectral data.
Finally, we qualitatively demonstrate that the multi-scale sinusoidal m/z embeddings capture high resolution structure by studying UMAP (McInnes et al., 2018) projections of the embeddings.
The results presented here represent the first direct empirical demonstration of the utility of sinusoidal embeddings to advance the state of the art of metabolomics research.
1. R ELATED WORK
Modeling numerical data in terms of sinusoidal functions has a long history in many scientific fields.
In machine learning, sinusoidal embeddings are most commonly used to encode the discrete positions of natural language token inputs in transformer models, which are otherwise position-agnostic (Vaswani et al., 2017).
Other work has used sinusoidal embeddings for multi-dimensional positional encoding in image recognition (Li et al., 2021a).
In mass spectrometry, sinusoidal embeddings have also been described in the context of modeling protein sequences from mass values, either with (Qiao et al., 1904) or without (Yilmaz et al., 2022) an initial mass binning step.
These techniques have never been applied in the domain of metabolomics.
This field considers small molecule chemistry where the objects of interest are 2 - 3 orders of magnitude smaller in mass and lack the sequential structure of proteins.
Consequently, the tasks and challenges of modeling of small molecules are sharply divergent from those stemming from proteomics.
For modeling m/z values in the field of metabolomics, previous machine learning models have relied primarily on discretization of the continuous mass inputs.
This is usually accomplished by binning m/z values into fixed length vectors with peak intensity as the value for each element.
Various authors have used binned representations of spectra for spectral library search
(Huber et al., 2021b), unsupervised topic modeling (van Der Hooft et al., 2016), and molecule identification (Kutuzova et al., 2021; Litsa et al., 2021).
Alternatively, masses have been tokenized via rounding for tasks such as unsupervised spectral similarity (Huber et al., 2021a) and molecule prediction from synthetic data (Shrivastava et al., 2021).
Other discretization approaches rely on tokenization of m/z values by direct molecular formula assignment.
Even when this method is able to uniquely identify molecular formulas for every fragment ion, it still faces a problem endogenous to tokenization: it’s very difficult to reason about the relationships between discrete tokens except by pattern recognition over a very large number of training examples.
Böcker & Rasche (2008) addressed this difficulty by modeling the fragmentation process with a tree structure, and Dührkop et al.
(2015) used these fragmentation trees as inputs to a molecular fingerprint prediction model that is used to assign molecules to spectra from molecular structure databases.
M ODEL 2. B ASE MODEL ARCHITECTURE
An MS2 spectrum,
[Equation]
is composed of a precursor m/z and a set of N fragment peaks at various m/z ’s and intensities ( I ).
Various other data are typically collected, including precursor abundance, charge, collision energy, etc , but these are not used in this work.
Transformer encoders (Vaswani et al., 2017) without the positional encoding are explicitly fully-symmetric functions and hence are ideally suited to model a set of fragmentation peaks.
We therefore take our base model, SpectrumEncoder ( S ) , to be a transformer encoder, whose inputs are a set of ( m/z, I ) pairs that includes the precursor along with all of the fragment peaks.
We normalize the intensities to a maximum of 1 for fragments and assign an intensity of 2 to the precursor.
Finally, we take as output the embedding vector from the final transformer layer corresponding to the precursor input.
See Figure 2.
Note, the use of transformers to study synthetic MS2 spectra has been explored in Shrivastava et al.
This flexible approach enables us to experiment with and compare various representations of MS data.
We pursue two approaches: tokenization of m/z and modeling m/z as numerical values via sinusoidal embeddings.
Before describing these, we need the following definition for a simple two layer feed forward MLP,
[Equation]
Note that each occurrence of FF we employ below is a separate instance with distinct weights.
2.1. T OKENIZED m/z PEAK EMBEDDING
Our first approach is to discretize m/z by rounding to 0 .
1 precision and treating these objects as tokens (as in an NLP context).
These tokens are embedded in a dense vector space as in Huber et al.
(2021a); Mikolov et al. (2013), via an embedding function TE .
We then construct the token peak embedding,
[Equation]
where ∥ denotes concatenation.
A diagram of PE token is shown in Figure 1a.
2.1. S INUSOIDAL m/z PEAK EMBEDDING
We now consider a numerical peak embedding that utilizes the full m/z precision.
We employ a sinusoidal embedding as follows:
[Equation]
The frequencies are chosen so that the wavelengths are log-spaced from λ min = 10 − 2 .
5 Daltons to λ max = 10 3 .
3 Daltons , corresponding to the mass scales we wish to resolve.
This embedding is inspired by Li et al.
(2021a)
; Vaswani et al.
Similarly to Equation 3, we construct the sinusoidal peak embedding,
[Equation]
We also experiment with casting the m/z inputs as half, single, and double precision floating point values.
Equations 4-5 are computed at the precision specified by m/z and the results are then cast to the precision of the model weights used for training.
A diagram of PE sin is shown in Figure 1b.
2.1. T RANSFORMER
The set of fragment peaks and the precursor are passed through either of the two peak embeddings specified above.
The result is a sequence of embedding vectors which is then passed through a transformer encoder:
[Equation]
[Figure]
Here PE stands in for either PE token or PE sin .
TransformerEncoder has embedding dimension d = 512 and six layers, each with 32 attention heads and an inner hidden dimension of d .
Note that all hidden and embedding layers in our peak embeddings above have dimension d as well.
As mass spectra have no intrinsic ordering, we opt to not include a positional encoding.
In order to get a single embedding vector as an output, the final transformer layer query only attends to the first embedding, corresponding to the position of the precursor m/z .
A diagram of our full model architecture is shown in Figure 2.
2. M ODEL TRAINING
We train the base models discussed here via two independent tasks aimed at applications within metabolomics and medicinal chemistry.
These tasks are described in detail below.
We build these models using PyTorch (Falcon, 2019; Paszke et al., 2019) and train them with the Adam (Kingma & Ba, 2014) optimization algorithm with parameters β 1 = 0 .
9 , β 2 = 0 .
999 , and learning rate α = 5 .
× 10 − 5 .
We also use a weight decay (Krogh & Hertz, 1991) parameter of 0 .
1 , a gradient clipping a parameter of 0 . 5 , and a dropout (Srivastava et al., 2014) parameter of 0 .
All models were trained using between 25 and 50 epochs.
Finally, all model weights were trained using half precision floating point values, regardless of the precision of the m/z inputs.
2.2. S PECTRAL SIMILARITY WITH S IAMESE NETWORKS
A common workflow for the analysis of mass spectrometry data is to compute some notion of spectral similarity for pairs of spectra that is intended to correlate with the molecular similarity of the underlying compounds (Yilmaz et al., 2017).
This can then be used for tasks such as spectral library search (Stein & Scott, 1994) and molecular networking (Quinn et al., 2017; Watrous et al., 2012).
Historically, most spectral similarity metrics used in the field have been heuristic-based (Li et al., 2021b;
Yilmaz et al., 2017).
More recently, spectral similarity methods that make use of techniques from deep learning have been developed (Huber et al., 2021a;b).
The latter of these, MS2Deepscore, is a Siamese network that trains a simple MLP (by binning m/z ) to predict a measure of molecular similarity (Bajusz et al., 2015) from pairs of mass spectra.
The loss function is the MSE between the ground truth molecular similarity and the cosine similarity evaluated on the dense embedding vectors generated by the base model.
Molecular similarity scores range from 0 to 1 .
The scores of randomly sampled pairs from our labeled dataset are not uniformly distributed and strong similarity matches are rare.
Huber et al.
(2021b), describe a procedure to sample pairs of spectra such that the molecular similarities will be uniformly distributed.
We use this procedure to sample pairs for training and evaluation tasks.
Following Huber et al.
(2021b)
we also train a Siamese network, but we do so using the base model transformer architectures outlined in Section 2.1 instead of with the binned mass vectors used by
Huber.
We report performance results for models trained using PE token and PE sin below in Section 3.2.
P ROPERTY PREDICTION
We also train the base model architecture to predict 10 chemical properties relevant to medicinal chemistry directly from an MS2 spectrum.
We do so by training the following network,
[Equation]
where the final layer has dimension equal to the number of properties we wish to predict.
We also scale all of our training labels to zero mean and unit variance.
For inference, we run predicted properties through an inverse-scaler.
As in Section 2.2.1, we train multiple versions of this property prediction model.
Due to the lack of existing methods for comparison, we also train a version of
Equation 8 on binned spectrum representations for a simple baseline.
We replace SpectrumEncoder with the MS2Deepscore feed forward architecture found in Huber et al.
(2021b)
for this baseline.
The properties we predict are standard indicators of druglikeness and bioavailability in the field of medicinal chemistry.
These include the properties that make up the Quantitative Estimate of
Druglikeness (QED) along with several others (Bickerton et al., 2012; Lipinski et al., 2012; Veber et al., 2002).
For the complete list of properties, see Table 2.
All properties are computed deterministically from chemical structure using RDKit (Landrum et al., 2013), so there is no additional dependency on experimental data.
R ESULTS 3. D ATA
We construct a set of labeled MS2 spectra by combining a number of publicly and commercially available datasets and one non-publicly available dataset (Wang et al., 2016; Mehta, 2020; Sawada et al., 2012;
Horai et al., 2010; Mikaia et al., 2014; Smith et al., 2005; Mardal et al., 2019).
We allow spectra collected in both positive and negative ion mode, and across a variety of instrument types, collision energies, and other important instrument parameters.
We constrain that spectra in our dataset have at least 5 peaks and at least 3 decimal places of m/z resolution.
We strip all stereochemistry from our molecular structure labels, which is a common step taken in MS2 modeling and allows for better molecule-disjoint splitting.
Chemical properties and fingerprints are then computed from the cleaned molecules using RDKit (Landrum et al., 2013).
Our resulting dataset has 1 , 251 , 830 spectra corresponding to 45 , 351 distinct structures.
In metabolomics experiments, biological samples contain both compounds that have previously been profiled in MS2 libraries and compounds that have not been profiled.
Identifying the structure and properties of molecules that are already in spectral databases is an important task on its own (e.g. “dereplication”). We therefore split our data in a manner that enables us to approximate model performance on both “known" and “novel" compounds.
We partition our set of labeled spectra into a training set, development set (i.e. validation set), and test set of respective sizes 1 , 214 , 812 , 18 , 750 , and 18 , 268 as follows.
We randomly select a set of 1 , 002 molecules present in our data and place all spectra associated with half of them into the development set, and all spectra associated with the other half into the test set. Of the remaining spectra, we place 998 randomly-selected spectra into the development set and 998 randomly-selected spectra into the test set. The remainder of the spectra are placed into the train set.
In this manner, a subset of each of the development and test sets is molecule-disjoint from the training set.
Henceforth, we refer to any spectrum in the development or test set as corresponding to a “known" molecule if that molecule is present in the training set, and a “novel" molecule if it is not.
We use the development set for hyperparameter optimization and only access the test set to compute reported metrics.
3. S PECTRAL SIMILARITY
A key application of any spectral similarity method is spectral library search (Stein & Scott, 1994) (or dereplication), whereby an unknown MS2 spectrum is identified by searching through a database of curated spectra obtained from known molecules.
To benchmark our various spectral similarity models, we query spectra from our test set against a lookup reference (which is the same as our training set).
Every query spectrum is matched, by maximum similarity, to a reference spectrum.
We then assess accuracy on a set of spectra queries by determining the fraction of matches that exceed a particular molecular similarity threshold.
We use two such thresholds, 0 . 6 for approximate match and 1 . 0 for exact match.
Finally, we typically have a varying number of recorded spectra per distinct molecule so all reported accuracies are macro averaged over query molecules.
Because novel molecules do not occur in the reference set, we estimate the maximum theoretical performance on the prediction task for novel molecules by evaluating the best molecular similarity matches between the novel test set and the reference set.
A spectral similarity model that perfectly approximates molecular similarity would score 0 .
89 on approximate matching.
We also evaluate the performance of a random baseline by uniformly sampling pairs of spectra and computing their corresponding molecular similarity.
Only 0 .
713% random spectra pairs from our data have a corresponding molecular similarity above the threshold for an approximate match.
We train a number of spectral similarity models, as described in Section 2.2.1, and will refer to the sinusoidal Siamese transformer and tokenized Siamese transformer models as the sinusoidal and tokenized models respectively.
For baselines, we train both a Spec2Vec (Huber et al., 2021a) and an MS2Deepscore (Huber et al., 2021b) model on our data.
For an additional baseline we also evaluate a current standard approach that uses a modified cosine spectral similarity (Watrous et al., 2012).
In Table 1 we report spectral library search accuracies (based on molecular similarity thresholds) for both known and novel compounds obtained from a number of spectral similarity models.
We omit exact match accuracies for novel compounds since these have been sampled so that no exact matches can be found.
In the same table we also report the MSE loss between predicted and actual molecular similarity on spectrum pairs drawn from train, known, and novel spectra sets.
No MSE can be reported for Spec2Vec as this method does not predict a molecular similarity.
We find that all considered learned approaches improve on modified-cosine spectral library search.
Of the learned approaches, we find that Spec2Vec has strong, only surpassed by the double precision sinusoidal model, spectral library search performance on novel, but only beats MS2Deepscore on known.
MS2Deepscore only outperforms the tokenized model on novel and underperforms everything on known.
Moreover, MS2Deepscore is substantially less accurate at predicting the specific pair molecular similarity on known and novel spectra when compared to all transformer approaches.
The double precision sinusoidal model produced the best performance on all evaluation tasks, beating all baselines.
This model produces state of the art known spectral library search accuracies of 0 .
937 and 0 .
97 for exact and approximate matching respectively.
Numerical representation of MS [Table] m/z via sinusoidal embeddings consistently outperforms all the discretization based approaches we benchmarked [Table]
To assess whether the multi-scale embeddings are learning from high resolution information [Table] we also train and evaluate versions of our sinusoidal model where m/
z ’s are cast to half precision floating point values.
Half precision floating point can only resolve up to approximately parts per O (10 , 000) , far below the parts per million precision of experimental mass spectrometry data.
The performance of the half precision model, reported in Table [Table] drops to be on par with the tokenized model (worse on known and better on novel).
We do not report results with single precision m/z since these were indistinguishable from double precision m/z .
On all evaluation metrics [Table] using m/z inputs of precision greater than 16 bits dramatically improves performance of the sinusoidal model.
From this, we draw the implication that sinusoidal embedding of m/z is able to effectively learn from with the wide range of scales present in high precision mass spectrometry measurements and this partially accounts for the superior performance of this method.
3.2. Q UALITATIVE EMBEDDING ANALYSIS
To further characterize the respective properties of token and multi-scale sinusoidal embeddings, we inspect UMAP (McInnes et al., 2018) projections of our m/z embeddings in Figure 3.
For this analysis, we use siamese transformer models described in Section 2.2.1.
For our sinusoidal models, we embed 50 , 000 m/z values between 0 and 1 , 000 Daltons using FF (SE ( m/z )) .
We do not use the full peak embedding function as we are not interested in the embedding of intensity information.
Because our tokenization procedure involves rounding m/z values to 1 decimal place, for the tokenized model we only embed 10 , 000 m/z values between 0 and 1 , 000 Daltons using the embedding function TE ( m/z ) .
While m/z measurements are continuous, the space they represent is inherently discrete.
Two molecular fragments differing in atomic composition could nevertheless have a very similar m/z ’s.
However, the large separation between molecular mass scales and the mass scale at which nuclear forces manifest themselves means that the relative atomic composition may be deduced from the fractional m/z (Jones et al., 2004; Pourshahian & Limbach, 2008), defined by
[Equation]
Therefore, we expect a quality embedding of m/z to embed fragments similar in both m/z and { m/z } close to one another, while paying greater attention to the latter.
As is seen in Figure 3, embeddings from tokenized and low precision sinusoidal models are able to capture general trends in m/z .
However, they fail to preserve distances in m/z and show little to no structure in { m/z } .
In contrast, the high precision sinusoidal embeddings allow our models to represent important information in { m/z } , while preserving distance in m/z . [Figure]
3. P ROPERTY PREDICTION
Many applications in metabolomics can be unblocked with knowledge of just a limited set of chemical properties, without need for identification of molecular structure.
Towards this end, we investigate the parallel task of chemical property prediction from MS2 spectra.
We evaluate sinusoidal vs .
tokenized m/z embedding performance on a list of properties that have compelling applications in drug discovery and are easily computable from our molecule labels.
For this experiment, we train a sinusoidal model and several baseline models as described in Section 2.2.2.
In Table 2, property prediction models are evaluated on known and novel structures using R 2 .
All transformer models outperform our feed forward baseline, illustrating the difficulty of the task and the significance of our results.
Half precision sinusoidal models produced an R 2 of 0 .
( 0 . 936 ) on known (novel) molecules averaged across all 10 properties.
These results are competitive with tokenization but substantially underperform when compared to higher precision sinusoidal embeddings.
When m/z values are specified at double precision, sinusoidal embeddings show improvements over tokenization on each property, resulting in an 11% improvement over tokenization on R 2 for novel molecules averaged across all properties.
This is further evidence that the multi-scale information captured by sinusoidal embeddings allows models to better understand the nuances of MS2 and chemical space.
In addition, the substantial performance of our sinusoidal property prediction model is strong enough to enable prioritization over a large number of unknown metabolites.
This has major implications in the drug discovery context, as inspecting the properties of unknown molecules in complex mixtures was previously not possible without isolation and individual experimentation.  C ONCLUSIONS
Through a number of experiments, we demonstrate that MS2 modeling methods that use a numerical representation of m/z via multi-scale sinusoidal embeddings are superior to methods which make use of discretized m/z .
In order to obtain insight into the performance attributed to sinusoidal
[Table]
quantitative estimate of druglikeness
[Table]
embeddings, we present evidence that indicates that sinusoidal embeddings enable learning from the high-precision data that is produced from MS2 experiments. The various results presented here represent the first direct empirical demonstration that the sinusoidal representation of (high-precision) m/
z s measured in modern tandem mass spectrometry experiments enables deep learning models to model MS2 spectra with greater fidelity.
These results include performance improvements on spectral library search and property prediction tasks in the metabolomics context.
Moreover, we present results that compare across numerical floating point precisions for the mass inputs, and show that sinusoidal embeddings perform better when higher mass resolution data is input. This suggests that neural networks with sinusoidal embeddings for the input masses are able to learn from high resolution mass data, which is not possible with standard binning approaches.
Finally, we demonstrate that high-dimensional structure of the mass embeddins emerges only when high precision mass values are utilized. The modeling work represented here produces state of the art results on spectral dereplication tasks.
We also produce strong results on ML chemical property prediction from MS2 spectra.
The property prediction results of are sufficient quality to functionally inform and advance drug-discovery efforts.
We expect that sinusoidal embeddings of m/z will be a critical component of further research that incorporates MS2 spectra modeling, and we look forward to exploring their use in other contexts.
R EPRODUCIBILITY S TATEMENT
We train our model on a combination of free public datasets (Wang et al., 2016; Mehta, 2020; Sawada et al., 2012;
Horai et al., 2010), commercially available datasets (Mikaia et al., 2014; Smith et al., 2005), and one non-publicly available dataset (Mardal et al., 2019). The necessary preprocessing steps are outlined in Section 3.1.
The model architecture is explained in detail in Section 2 and all parameters used in model training are specified in Section 2.2. We include our source code in the supplementary materials.