3.1 Dataset Prepare 16 3.2 Feature extraction 16 3.2.1 Energy or scoring function (7 features) 16 3.2.2 Protein secondary structure (5 features) 16 3.2.3 Solvent accessibility (3 features) 18 3.3 Parameters Optimization of Machine Learning Methods 18 3.3.1 Linear Model 19 3.3.2 Decision Tree 19 3.3.3 Neural Network 19 3.3.4 Boosting 20 3.3.5 Random Forest 20 Chapter 4: EXPERIMENTAL Results 21 4.1 Data Set 21 4.2 Evaluation Parameters 22 4.2.1 Pearson’s correlation 22 4.2.2 Quality score 23 4.2.3 Quality lost 23 4.3 Features’ Correlation Coefficient 23 4.4 CASP10 model 26 4.4.1 Linear Model 30 4.4.2 Decision Tree 31 4.4.3 Neural Network 33
Abstract Computational protein structure prediction is one of the most important problems in bioinformatics. In the process of protein three-dimensional structure prediction, assessing the quality of generated models accurately is crucial. Although many model quality assessment (QA) methods have been developed in the past years, the accuracy of the state-of-the-art single-model QA methods is still not high enough for practical applications. Although consensus QA methods performed significantly better than single-model QA methods in the CASP (Critical Assessment of protein Structure Prediction) competitions, they require a pool of models with diverse quality to perform well. In this thesis, new machine learning based methods are developed for single-model QA and top-model selection from a pool of candidates. These methods are based on a comprehensive set of model structure features, such as matching of secondary structure and solvent accessibility, as well as existing potential or energy function scores. For each model, using these features as inputs, machine learning methods are able to predict a quality score in the range of [0, 1]. Five state-of-the-art machine learning algorithms are implemented, trained, and tested using CASP datasets on various QA and selection tasks. Among the five algorithms, boosting and random forest achieved the best results overall. They outperform existing single-model QA methods, including DFIRE, RW and Proq2, significantly, by up to 10% in QA scores.
Percentage of secondary structure 0.3020 0.3125 Percentage of Helix 0.2744 0.2729 Percentage of Sheet 0.2431 0.2003 Percentage of Coil 0.0148 0.0256 Consistence Score of Secondary Structure 0.3164 0.3214 Percentage of matching Solvent Accessibility 0.3351 0.3562 Matching of Bury Amino Acid 0.2721 0.3087 Matching of expose Amino Acid 0.1726 0.1558 DDFIRE 0.3171 0.3142 DFIRE (too many Inf) 0 0 DOPE 0.2708 0.2465 OPUS_CA 0.3642 0.3518 Proq2 0.3292 0.3200 RAPDF 0.3087 0.3240 RW 0.2585 0.2838 Based on the correlation coefficient results showed above, we can see that in CASP10 all model dataset (dataset one) and QA stage one dataset (dataset two), secondary structure based features, solvent accessibility based features and Proq2 showed better performance. The common ground between these two datasets is that they all contain the knowledge of all the models from different servers. That means they include good, medium, and bad models in the datasets. Likewise, structure based features and Proq2 work well on jagged-quality models pool. QA stage two dataset consist of 150 best models in dataset one, which means most of the models in this dataset got a better quality. When tested on QA stage two dataset
The Pearson’s correlation of linear model (0.5716, 0.5862) is 12 percent higher than Proq2, but still lower than consensus-based method. Actually, almost all of those targets are worse than consensus 4.4.2 Decision Tree Figure 4.4.5 Compare the result of decision tree with Proq2 on dataset one Figure 4.4.6 Compare the result of decision tree with consensus-based method on dataset one When tested on dataset one, the performance of decision tree is comparable with Proq2, for some of the targets Proq2 got a very bad result, decision tree got much better result. But on both quality lost and Pearson’s correlation, decision tree is obviously worse than consensus-based method, and only a few targets got a lower quality lost. 4.4.3 Neural Network Figure 4.4.7 Compare the result of Neural Network with Proq2 on dataset one
0.0537). Among all of the 103 targets, 51 (54) got lower GDT-TS (TM-score) lost using linear model. The Pearson’s correlation of linear model (0.4083, 0.4062) is about 8 percent higher than Proq2, but still 10 percent lower than consensus-based method. Among all of the 103 targets, 38 (38) of the targets got a higher Pearson’s correlation than consensus. 4.6.2 Decision Tree Figure 4.6.5 Compare the result of Decision Tree with Proq2 on dataset three Figure 4.6.6 Compare the result of Decision Tree with consensus-based method on dataset three When tested on dataset three, decision tree got a quality lost (0.0640, 0.0635) which is worse than Proq2 (0.0566, 0.0549) and consensus-based method (0.0541, 0.0537). Among all of the 103 targets, only 33 (43) got lower GDT-TS (TM-score) lost when using decision tree. The Pearson’s correlation of decision tree is still much worse than Proq2 and consensus-based method. Among all of the 103 targets, only 20 (20) of the targets got a higher Pearson’s correlation than consensus. 4.6.3 Neural Network
decision tree, random forest uses large number of trees to overcome the overfitting problem and when the tree number became larger, the performance became much more stable. Boosting also make the performance of algorithm stable and better by fix the initial model with the shrunken version of the following trees. These two methods also showed strong ability on selecting the best models directly. Among these five methods, linear model and decision got the smallest system cost, and it is very fast. Neural Network is more complex but the procedure is not time consuming. Comparing with them, boosting and random forest will generate a very large model which can arrived a few gigabytes, and consume much more time for generating large number of decision trees. In this study, even though different machine learning methods showed really good performance, more feature work is still needed. The first problem is some of the applications used to generate features are not the newest version, by update these applications, the quality of features may be improved. The second issue is the tree number of random forest and boosting just reached 3,000 and 1,000 in this work, but in some other researches [36], this parameter is tested up to 10000. Especially when testing boosting, it still showed a tendency of decreasing on quality lost when the tree number grew. Another important problem is that though random forest and boosting result in a lower correlation than consensus-based method, they still got a better performance on quality lost, this may hints machine learning methods is more suitable on some kinds of special targets or models. And on some targets that consensus-based