How to do an evaluation: pitfalls and traps

The recent literature is replete with papers evaluating computational tools (often those operating on 3D structures) for their performance in a certain set of tasks. Most commonly these papers compare a number of docking tools for their performance in cognate re-docking (pose prediction) and/or virtual screening. Related papers have been published on ligand-based tools: pose prediction by conformer generators and virtual screening using a variety of ligand-based approaches. The reliability of these comparisons is critically affected by a number of factors usually ignored by the authors, including bias in the datasets used in virtual screening, the metrics used to assess performance in virtual screening and pose prediction and errors in crystal structures used.
Introduction
Based on the large number of papers recently published, it has become obvious that a large proportion of the computational chemistry community, both in academia and in industry, is very interested in evaluating and comparing software for a number of different purposes. A large number of publications have appeared over the last 5 years or so that are focused on the evaluation of docking tools for pose prediction [1], virtual screening [2] and affinity prediction [3]. There have also been a number of recent publications examining the performance of ligand-based tools in similar tasks. The ligand-based tools have also been evaluated in the areas of pose reproduction (by conformer generators [4–6]), virtual screening [7] and affinity prediction [8]. In the following sections some issues with studies on pose prediction and virtual screening will be discussed.
Pose prediction
A common method of evaluating a docking program is to gauge its performance in cognate re-docking or self-docking. In this process a ligand is extracted from a co-crystal structure with its target protein and the program is challenged to pose the ligand as closely as possible to its experimentally identified structure. It may be argued that cognate re-docking is not a task commonly faced in the normal use of docking tools, since cross-docking (docking of a ligand into a structure with which it was not crystallised) is the actual application of a docking tool [9]. However, the exercise remains popular, doubtless in part due to the relative ease of execution of a self-docking study and partly for comparison with previous studies. As has been pointed out previously [10], comparing docking programs for their ability to predict the bioactive pose of a ligand is difficult for a number of reasons, some of which are obvious while others are subtler. However, the operational difficulties in comparing programs in a robust way should not cause other sources of error to be ignored. It is a truism that it is meaningless to compute a property with greater precision than the accuracy of the experiment that measures that property. Unfortunately, as will be seen in subsequent sections, this is often ignored by authors of papers in the area of pose prediction.
When comparing tools for pose prediction, the heavy atom root mean square deviation (RMSD) between the computed and experimental poses is the de facto standard. A regrettably common method to compare docking tools for pose prediction success––illustrated, for example, in the papers presenting results from the MolDock program [11] or the Glide XP evaluation [8]––is to compare the average RMSDs across a set of structures. The use of the average RMSD admits of many possible problems of interpretation, not least of which is the biasing of the average by a few very large or very small numbers. In the case of the MolDock results, the mean RMSD across a set of structures was used to suggest that the performance of MolDock was comparable to that of GLIDE and superior to that of Surflex (see Table 1). However, the use of the median RMSD, which is far less biased by a few extrema, suggests a different conclusion—that MolDock is in fact somewhat superior to Surflex, but not as good as GLIDE. For a similar analysis on the drawbacks of using average RMSD as a comparator, see Cole et al. [10]. A number that is noticeably absent from this, and all other comparisons of docking tools using RMSD, is an error bar on the average RMSD (which can be calculated by bootstrapping). Without such an error bar it is impossible to assert that any tool compared in this study is actually better than any other.
Table 1Comparison of RMSD results from a set of docking enginesMolDockGLIDESurflexMean1.381.381.86SD1.491.742.02Median0.920.691.10
A separate issue with these two experiments involves contamination of the dataset used to evaluate the performance of the tools. In the case of the Glide results the RMSD’s given are not to the deposited crystallographic pose but rather to one that results from a pre-processing step, as noted by the authors of the MolDock paper [11]. As such this is not an “apples to apples” comparison, since the Glide pre-processing step optimizes protein and ligand coordinates using the force-field component of the Glide scoring function, which necessarily introduces bias in the structure. In the MolDock case the authors have essentially trained the MolDock fitness function on the 77 complexes that they use to evaluate its performance. As such the reported results give no indication of the likelihood of success in predicting a pose for a system upon which MolDock was not trained.
The RMSD between two poses is a geometric measure, comparing the atomic positions between the experimental structure and the docked or predicted structure. Other metrics based on comparing the geometry of the experimental and the computed pose have been developed, such as relative displacement error (RDE) [12]. These, and all other atom-based metrics, suffer from the drawbacks pointed out by Cole et al. [10]. A more serious problem for metrics like RMSD and RDE is that they attempt to indicate the quality of reproduction of a model for the data, not the crystallographic data itself, i.e., the electron density. This disconnection between RMSD as a metric of quality for pose prediction and the original crystallographic data has been a cause for concern. Some attempts have been made to arrive at metrics that better reflect the reproduction of the actual crystallographic data, in particular real-space refinement (RSR) [13]. Unfortunately, RSR has not been widely used, possibly because it is more difficult to calculate than atom-based metrics like RMSD or RDE. Other metrics that are not based purely on atom position, such as interaction-based accuracy classification (IBAC) [14], try to reflect the ultimate use of the predicted pose, i.e., determining the nature of the interactions that the ligand makes with the protein. While the IBAC approach has value in that it assesses a computed pose by its interactions with the protein, it is not amenable to automation; it is therefore tedious to assemble sufficient data to make statistically robust comparisons between tools based on IBAC.
However, the problem of choosing which metric to use to compare pose prediction studies is dwarfed by the difficulty in choosing a dataset of protein–ligand co-complexes upon which to perform the comparison. A widespread tendency in conformer reproduction and pose prediction studies is to ignore even the possibility of error in the crystal structures that are being reproduced. Crystal structures are often treated as perfect, infinitely precise and accurate representations of the atomic details of a protein–ligand complex. There are a number of reasons why this is not so; a few will be discussed in the following paragraphs.
Crystal structures are models
This statement makes up the warp and weft of crystallography, yet in the transition of crystallographic data from crystallographer to computational chemist the distinction between the actual data and a model for that data is often lost. The actual data in crystallography is, of course, diffraction data leading to electron density. The atom positions that make up a crystal structure are a model that attempts to explain this data in as complete a fashion as possible. This is illustrated in Fig. 1.
Fig. 1Fitting atoms into electron density produces a crystallographic model
The process of fitting or modeling the atoms (of a ligand or of a protein) into the electron density is not always straightforward, and the problems are often more serious when the fitting of a ligand into density is being carried out. The sources of these problems include:Incomplete or fragmentary density;The electron density not defining the positions of all atoms unambiguously;Poor structural parameters are used for the fitting process, which can give inappropriate conformations (particularly of ligands);Errors by the users, arising from careless treatment of the data or lack of expertise with small molecules.
A variety of metrics can be calculated for a given crystal structure that attempts to give an indication of the quality of the model. While no single number can encompass the quality of a structure, some of the metrics of quality are more useful than others. The most commonly used is the nominal resolution. This is a measure only of the quantity (of data collected) and not of the quality of the data nor, most especially, of the quality of the model fitted to that data. (For a discussion on the problems with resolution, see Ref. [15]) It is therefore unfortunate that nominal resolution is often the only criterion used to select a protein structure, and that its meaning is often misconstrued. For example, in a paper [16] by Nissink et al. the following incorrect statement is made: “The resolution of a protein structure is directly related to the accuracy of the data.” Not only is this statement not entirely true, it also confuses what the nominal resolution means (quantity of data collected) with the accuracy of a model, which are two very different things. A metric that does attempt to assess the quality of the fit of the crystallographic model to the source data is the so-called Rfree, introduced by Brunger [17]. Rfree is an indication of how well an atomic model explains a small percentage of the density data that was omitted during the fitting process, and is thus an unbiased metric that can be reliably used to distinguish a well fitted model from a poorly fitted model. Unfortunately, Rfree is infrequently used as a metric of quality when selecting crystal structures for docking studies or other purposes (see Ref. [18] for the use of Rfree as a criterion for selection of structures). More frequent use of Rfree as one of a set of metrics for selection of crystal structures could help to avoid the selection of poorly fitted models for pose reproduction.
Crystal structures have unavoidable imprecision
As already discussed, nominal resolution is not a metric of quality for a structure and although Rfree indicates the quality of the fit of the model to the original data it can provide no estimate of the uncertainty in the atomic positions within that model. It is often assumed that the experimental uncertainty in the atomic positions in a crystallographic model can be estimated by use of the isotropic B-factor, which is supposedly a measure of the thermally driven fluctuation in atomic positions. However B-factors are a refined parameter and so they cannot be compared between structures without detailed knowledge of the restraints used [19]. It is therefore impermissible to use low B-factors as a sure indication of low positional uncertainty. Nor is it possible that there exists a uniform cut-off for B-factors that indicates low positional uncertainty in all structures, though such an assumption is frequently made [20]. Another potential problem with B-factors is that they can be over-refined in the pursuit of better quality metrics for a structure. An example is given in Fig. 2, which shows the ligand (leucinol) from the structure 5ER1 from the RCSB [21] annotated with its B-factors. The B-factors for every atom in the ligand are <1. In contrast, B-factors even for well-located atoms in high quality models, are generally not <5 [22]. In this instance the crystallographer has over-refined the B-factors of the ligand atoms in order to improve the quality of the overall model. In this model the B-factors are unphysically low only as a consequence of a pathology of the refinement process. Accordingly, B-factors as an indication of local mobility in a protein structure should be treated with caution, as they are not simply an experimentally derived quantity but are free parameters in the refinement process.
Fig. 2B-factors for the ligand in the 5ER1 crystal structure
An alternative to using the average B-factor for estimating the uncertainty of the position of atoms an atom in a structure is the average coordinate precision (or diffraction-component precision index, DPI). The DPI expresses the average precision for the atomic coordinates in a protein structure [23] and as such can be used as a measure of the experimental precision of the atomic positions in that structure. The original formulation of DPI was very complex and has recently been recast in a more easily calculable way by Blow [24]. The use of DPI as a metric of quality for crystal structures used for docking studies was introduced to the computational community by Goto et al. [18], and their formula is given in Eq. 11.
In this treatment the standard error of position, σ(r), is related to the number of atoms in the unit cell, Natoms, the volume of the unit cell, Va, the number of crystallographic observations, nobs and the Rfree. It should be noted that the formula presented by Goto et al. is not precisely the same formula that Blow derives in his paper. In Eq. 1 the prefactor is given as 2.2, while in the original work by Blow the prefactor is given as 1.28. This is because Blow is calculating coordinate error for a particular axis, σ(x, y, or z), while Goto et al. are calculating the error in the distance, σ(r), giving rise to the √3 difference between the two prefactors. As such it is appropriate to consider the error in the coordinates, σ(x, y, z) as a measure of the uncertainty in the atomic positions, and the error in inter-atomic distances, σ(r), when comparing a computed and an experimental structure.
The resolution of a crystallographic model, as has been mentioned, is often used to select protein structures for pose prediction by docking or conformer generation studies, on the assumption that resolution imparts some information on the quality and precision of the model. The DPI is a much more direct estimate of the reliability of crystallographic models when it comes to comparing experimental and computed atom positions (as is done in conformer reproduction or pose prediction). It is therefore of interest to compare the nominal resolution for a large number of “good quality” crystal structures with the DPI (σ(r)) for the same structures. A good dataset for this comparison is provided by the extensive investigations performed by Kirchmair et al. [4]. Here 776 co-crystal structures were used to provide experimental ligand structures that were then compared to sets of computed conformations from conformer generation applications. For 556 of these crystal structures there exists sufficient data to allow the DPI to be calculated and the relationship between the nominal resolution for these structures and their DPI is shown in Fig. 3. It is obvious from Fig. 3 that the statement by Kirchmair that “0.5 Å approximately represents the accuracy of protein X-ray crystallography” is not supported by the actual properties of the crystal structures they studied. In fact, in those cases where the DPI can be calculated, almost 56% of the structures from their paper have DPIs > 0.5 Å.Fig. 3Coordinate error for 556 structures from the paper by Kirchmair et al. [4]
Figure 3 illustrates a number of other interesting points. While the expectation that greater coordinate precision will arise from structures with better nominal resolution is generally borne out by the data, there are many exceptions. Table 2 shows some examples of structures where the nominal resolution gives an unexpected estimation of coordinate precision. In the top half of the table are structures with good nominal resolution but unexpectedly high DPI, while the lower half of the table shows some structures with low nominal resolution and either unexpectedly low or unexpectedly high DPIs. Accordingly, simply using nominal resolution as a metric of quality for structures to be used in a pose prediction or conformer generation study is insufficient to guarantee that structures of appropriate quality will be used.Table 2Resolution and DPI for selected structures from the Kirchmair datasetPDB codeResolution (Å)DPI (Å)1FC71.380.691FDO1.380.601JJT1.81.371JJE1.81.251CIB2.55.541ILH2.760.141C8M2.80.181QJX2.80.25
With the DPI for a structure in hand one can set a lower limit on the precision with which a computed conformation can reproduce an experimental one—the RMSD between the two conformations cannot be less than the DPI for the experimental structure. It can be seen that over half (55.6%) of the structures in this set has DPIs > 0.5 Å, while Kirchmair et al. report pose reproduction statistics both at <0.1 Å and at <0.5 Å RMSD. Since in over half of the structures in this dataset the DPI is >0.5 Å, Kirchmair et al. report pose reproductions at <0.5 Å RMSD that are more precise than the accuracy of the source data allows. This analysis is made on the conservative assumption that the error in the atomic positions in the computed pose is zero. In the Goto et al. paper [18] the assumption is made that the errors in the computed pose are the same as for the experimental pose. In this analysis a computed and experimental pose must be different by an RMSD of √2 × DPI for the difference to be significant, which would mean that an even higher proportion of the poses in the Kirchmair set have been reproduced with a greater precision than the experimental accuracy.
The same tendency to reproduce experimental data with a precision greater than the experimental accuracy is seen frequently in pose prediction experiments with docking engines. In Fig. 4 nominal resolution is plotted against the DPI for crystal structures from two well known docking validation sets, those for GOLD [25] and GLIDE [3]. The two graphs are plotted on the same scale to allow direct comparison. On each graph is also plotted an estimate of the theoretical lower limit for the atomic precision (the pink line). A formula describing the relationship between nominal resolution and DPI is given in Eq. 22 below, based on a derivation by Blow [24].Fig. 4The nominal resolution versus the coordinate error for a subset of the Gold (structures with resolution <2.5 Å) and the Glide data sets
The variables found in Eq. 2 are as follows: s is the percent solvent present in the crystal, Vm is the asymmetric unit volume to molecular weight ratio, C is the completeness of the data, and dmin is the nominal resolution of the structure. The ideal lines shown in Fig. 4 were calculated using Eq. 2 and assuming an s of 0.0, a Vm of 2.5, a C of 100%, and that Rfree is equal to the resolution/10.
Inspection of Fig. 4 shows that the GOLD set contains one structure (1YEE) whose calculated DPI lies below the theoretical lower limit. This is probably due to a mistranscription error in the PDB file. The PDB file for the 1YEE structure gives the number of reflections as 77209, while the number of unique reflections is 21342—a disparity that cannot be reconciled by reference to the data redundancy for this structure. Accordingly the calculated DPI for 1YEE is too low because the reported number of reflections is erroneously high. In the GLIDE set all structures have DPIs higher than our predicted theoretical minimum. In fact, in this set 52% of the structures have a DPI > 0.5 Å, and in 31% of the cases the reported RMSD for redocking is less than the DPI (so that the prediction is more precise than experimental accuracy allows). This is only one example of a publication in which protein structures are assumed to be free of uncertainty in their atomic positions; the literature abounds with others.
The combination of Rfree and the DPI for a structure can give a good overall picture of the quality of a model and the reliability of the atom positions within that model. In spite of the availability of these and other measures of quality, there are a number of test sets of protein co-crystal structures used for evaluating docking engines that appear to have been selected by other criteria that do not relate in any way to their quality [26]. Accordingly, the results from these studies should be treated with some caution.
Crystal structures have avoidable errors
The Rfree and the DPI are global measures of quality; other, local, measures of quality are also useful. For example, if a small portion of the atoms in a structure have been poorly fitted to the density, this will not be revealed by any global measure of fit. Only a local measure would reveal the error. A relatively common problem in co-crystal structures in the PDB is poor fitting of small molecules to the density, giving unrealistic ligand structures. These poorly fitted ligand structures are then used as a “standard” to judge the quality of a docking program’s or conformer generator’s performance. While the poor quality of some ligand structures in PDB models has been known for some time [27], these reports have been anecdotal and few systematic attempts to avoid such poorly solved structures (other than visual inspection) appear to have been undertaken. If appropriate attention is not paid to selecting good quality ligand structures, then a dataset could be constructed that contains ligand structures that have significant errors. For pose prediction studies, poorly solved ligand structures in the dataset must be avoided. It is senseless to try to computationally reproduce an “experimental” ligand structure that has been incorrectly fitted to the electron density. Examples of some possible errors in ligand structures that should result in structures that are not computationally reproducible are shown in Figs. 5 and 6.Fig. 5Ligand conformation from 1A8T structure. The conformation has two serious atom–atom clashesFig. 6Ligand conformation from the 1A4K structure. The cis-amide group is an error of fitting
In the case of the 1A8T structure (Fig. 5), the deposited ligand coordinates contain two serious atom–atom clashes that give the resulting conformation very high energy (28.4 kcal/mol above a refitted structure by the MMFF94 forcefield [28]). The deposited coordinates are clearly in error in this case and no docking engine or conformer generator should be expected to reproduce such a structure.
The error shown in Fig. 6 for the 1A4k structure is of a more subtle nature. Here the crystallographer has fitted a highly strained cis-amide into the electron density with no compelling reason from the electron density to do so. The amide group is packing against a tyrosine residue from a symmetry mate in the unit cell, and makes no polar interaction with it. The corresponding trans-amide, which fits the experimental density just as well, is 15.5 kcal/mol lower in energy (using the MMFF94 force-field). Also this trans amide is able to make a hydrogen bond (with a backbone carbonyl group) that is not available to the cis conformation. Once again, this model should not be reproduced by a docking engine or conformer generator, as it is obviously not the correct solution.
One of the only studies that bears on the issue of ligand strain was published by Perola and Charifson in 2004 [29]. The authors examined ligand strain in a number of public structures from the RCSB database and proprietary structures from Vertex internal collection and found that 10% of the ligand structures examined had high strain (10 kcal/mol or greater above the global minimum). The two examples discussed above, 1A8T and 1A4K, are both part of the Perola dataset. Clearly these two ligand structures have high strain energies due to errors in fitting and not due to a fundamental property of the ligand’s conformation in complex with the protein. Close examination of the rest of the models for the ligands (where structure factors are available) showed that a number of them are incorrect, and re-solving them provided lower energy structures in about 85% of the cases [30]. Recent re-examination of the Perola dataset by the Snyder group has provided further insights into the large strain energies originally reported [31]. It is therefore most likely that the vast majority of ligand conformations in complex with a protein show strain energies less than 6 kcal/mol, unless the ligand is rather large [32]. Those ligand conformations with strain energies higher than 10 kcal/mol are almost certainly incorrect. As such low ligand strain should be among the criteria for selection of structures for pose prediction, as structures with high strain are very likely to arise from errors in the fitting process.
It is clear from the foregoing discussion that, while crystal structures are an invaluable source of information on protein–ligand binding, these structures are not without many sources of confounding errors. These errors, those inherent to the data and the process of fitting, as well as those introduced by human error or the insufficiencies of the fitting program, should be borne in mind before using crystal structure data. The selection of a reliable set of structures for pose prediction is a therefore not a trivial task. An excellent study considering these issues can be found in Hartshorn et al. [33].
Virtual screening
Virtual screening can be defined as any method that ranks a set of compounds by some score. Successful virtual screening relies on having a scoring method that assigns good scores to interesting molecules (usually defined as active against a target protein of interest) and worse scores to uninteresting (inactive) molecules. Accordingly a successful virtual screen will provide, from the top of this ranked list, a set of compounds for experimental screening that is highly enriched in active molecules. This topic has been of great interest both in academia and in the pharmaceutical industry in recent years, and a large number of publications have appeared on the subject. While a few of the publications have investigated virtual screening conducted prospectively [34–36], the vast majority have been concentrated in the area of retrospective virtual screening. In the rest of this article we shall concern ourselves solely with the retrospective experiments. The goal of such experiments is often to identify an application that performs well on a given target, or across a wide range of targets, with a view to utilizing this application in prospective virtual screens.
There are a number of approaches to quantitating the success of a particular tool for virtual screening. The most often used, and simplest to calculate, is enrichment at a given percentage of the database screened. Enrichment (EF) is defined according to Eq. 33 (Hitssampledx% = number of hits found at x% of the database screened, Nsampledx% = number of compounds screened at x% of the database, Hitstotal = number of actives in entire database, Ntotal = number of compounds in entire database).
Enrichment appears to measure the quantity of most interest to those performing virtual screening: the ability of a tool to place a large proportion of the active compounds at the top of the ranked list. Enrichment is also simple to calculate and understand, so it seems the ideal metric with which to compare tools for their virtual screening performance. However, enrichment suffers from a number of significant drawbacks, especially when comparing results between studies or using enrichment to predict future performance:It is dependent on the structure of the dataset, in that datasets with larger proportions of actives will have a narrower range of possible enrichments.It penalizes ranking one active compound above another.It exhibits pernicious behaviour at the cut-off at which the enrichment is calculated.It gives no weight to where in the ranked list a known active compound appears. Thus to calculate enrichment at 1% in a virtual screen of 10,000 compounds, the number of actives (N) in the top ranked 100 compounds is needed. However the enrichment at 1% is the same whether the N active compounds are ranked at the very top of the list or at the very bottom of the top ranked 100.It is difficult to calculate analytically errors in enrichment, and there is no available literature for such a calculation.
With regard to point (i), experiments performed on different datasets cannot be compared when using enrichment, as the dynamic range of enrichment will be different for different datasets, but there are several cases where this has been done [37]. Early enrichment can also suggest overly impressive performance for tools that rank a small fraction of the active molecules very early in the list, but fail to give good ranks to the majority of the actives [37]. It has been shown by Seifert [38] that enrichment does not detect significant pathologies in the ranking function. Other metrics have been developed by some groups specifically to address some of these problems: RIE by Merck [39], cumulative probability by Molsoft [40] and average number of outranking decoys by Schrodinger [3]. These metrics have, historically, only been used by the groups that invented them and are thus not useful for comparison to studies conducted by other groups. This lack of direct comparability across studies is compounded by the fact that these metrics cannot be converted into a more commonly used metric that could be used to compare results. Also, as with enrichment, it is unknown how to estimate analytically errors in any of these metrics.
A metric used to determine success in detecting a signal in a background of noise is the receiver operator characteristic (ROC). The ROC curve is derived by plotting noise (fraction of false positives) on the x-axis versus signal (fraction of true positives) on the y-axis. The area under the ROC curve (AUC) is a widely used measure in a variety of fields including medical statistics, criminology and bioinformatics [41, 42]. When applied to virtual screening the ROC illustrates success in ranking actives (signal) above decoys (noise). The AUC for the ROC curve shows performance of a given tool when screening across the entire database is examined, not just at fixed, early points in the screen as enrichment does. The theoretically perfect performance of a virtual screening application gives the maximum area under a ROC curve (1.0), while random performance of a tool gives an AUC of 0.5. Areas under the curve of less than 0.5 imply a systematic ranking of decoys higher than known actives. For recent applications of the ROC curve in virtual screening, see [2, 43]. The AUC for the ROC curve is also known as the ‘discrimination’. Discrimination is defined as the fraction of occurrences that a randomly chosen true positive (active) is given a better score than a randomly chosen true negative (decoy). This number then allows prediction of the likely effectiveness of a tool in experiments that have not yet been conducted. This predictive ability is not provided by metrics such as enrichment, cumulative probability and average number of outranking decoys, because while the ROC describes a property of the application studied, the other metrics essentially describe a property of the experiment. The AUC assesses virtual screening performance across the entire database and many practitioners of virtual screening are, rightly, most concerned about early performance of the tools they use. This is one reason why enrichment is so commonly used to measure success. The metric of early performance based on the ROC curve is the true positive rate at fixed false positive rates. The true positive rate at a false positive rate of, for example, 1% is a much more robust measure than the enrichment at 1% and provides similar information about the early performance of a tool.
The AUC also offers the advantage that a statistically robust estimate of its errors can be estimated analytically from the AUC itself, using a method developed by Hanley [41]. This is not a property possessed by enrichment, average number of outranking decoys etc. For these metrics errors can only be estimated (by bootstrapping or other approaches) from the raw data, which is rarely provided. The error in the AUC, as with other metrics, is reduced by increasing the number of positives (active compounds) and by increasing the number of negatives (inactives). The Hanley treatment shows that the error in the AUC is most significantly reduced by increasing the number of actives, while increasing the number of decoys has a much smaller impact on the error. Therefore virtual screening datasets with high proportions of active compounds will provide results with lower error bars for the AUC. With the error for an AUC available, meaningful comparisons can be made between two or more different tools. However the other metrics used for virtual screening mentioned above do not allow an analytical estimation of their errors. Comparisons designed to determine which tool is superior for a given purpose, that are based on metrics assumed to be free of errors, are fraught with difficulty.
While the choice of metric may affect the relative ranking of tools compared on the same datasets, the composition of those datasets has a very profound effect on the results generated. Until very recently it has been common practice to assemble a dataset for virtual screening by seeding a set of active molecules against a target of interest into a background of compounds (decoys) chosen essentially at random. These decoys compounds were often drawn from public sources such as vendor catalogues. An example of this approach is the seminal paper on virtual screening by docking from the Rognan lab where decoy compounds were selected at random from the publicly available compounds [44]. This set of decoys, or a subset thereof, has been used extensively since its publication in 2000 [2 and references therein] so that this same set of decoy compounds has been used in more than 45 different published virtual screening experiments on a wide variety of target systems. Given the huge variety in the types of active molecules camouflaged in this same set of decoys it seems intuitively obvious that in some cases the active compounds for a given target will be very easily discriminated from these decoys. As such a number of the virtual screening experiments performed using this set of decoy compounds have given good results purely due to differences in simple properties between the actives and the decoys (vide infra). In the Rognan dataset there were 10 active compounds for each of the targets studied and 990 randomly selected decoys. There are two problems arising from constructing a dataset in this way. The first is that the small number of active compounds means that the errors can be very high (vide supra). The second is that trivial property differences between the active compounds and the decoys can result in undeservedly good performance. The first issue, low prevalence, is still widespread in retrospective virtual screening studies. Although low prevalence reduces the reliability of the results, many virtual screening experiments are still conducted using very low numbers of active compounds. Reasons for this could include a desire to mimic “real” HTS experiments, where hit rates are often on the order of 0.01–0.1% [45] or to allow enrichment, the most commonly used metric for success in these studies, the maximal dynamic range. Tribelleau et al. [43] show that the dynamic range of enrichment, the difference between random and maximal performance, decreases as the proportion of active molecules in a dataset rises. For a recent example of a retrospective virtual screening study with deliberately large numbers of actives, providing high statistical power and small error bars, see Ref. [37].
The second issue, systematic differences in simple properties between decoys and actives in retrospective virtual screening experiments, is much more serious. As has been pointed out in a number of publications, scoring functions in docking programs, which are almost always additive, are sensitive to molecular size or heavy atom count, the number of hydrogen bonds that the molecule can make etc. Accordingly, systematic differences in these simple molecular properties between actives and decoys will cause systematic differences in ranking. For example, active compounds with higher average heavy atom counts will tend to rank better than the decoys when scored by a function that is sensitive to heavy atom count. For a fuller discussion of this issue see Verdonk et al. [46]. An example of the influence of the selection methods for decoys on virtual screening performance is shown in Fig. 7. In this figure, two retrospective virtual screening studies against CDK-2 using the docking tool FRED [47] are compared using ROC curves. In both cases the same actives were docked against the same co-crystal structure, while different decoy compounds were used. In one case (the green line), the decoys were chosen at random from the Maybridge compound collection [48], in the other (the red line), decoys were chosen to match the properties of the actives based on simple 1D properties. The striking difference, especially in early performance, is obvious. Clearly the performance of FRED is heavily affected by the nature of the decoy compounds, and to obtain a predictive indication of the utility of FRED in virtual screening decoy sets similar to those giving the red line should be used. It is worthy of note that the AUC’s for the two experiments shown are different by more than their respective 95% confidence intervals, so that this difference is indeed statistically significant.
Fig. 7Effect of decoy selection method on virtual screening by docking
Another example of how much of the signal that separates active from inactive compounds arises from systematic differences between their properties is shown in Fig. 8. In these eight examples, drawn from the Surflex-Dock dataset [2], the performance of three 3D virtual screening methods––Surflex-Dock, ROCS [49] and FRED––is compared to a simple 1D method. In this 1D approach [46, 50], compounds are ranked by distance in an Euclidean property space to the centre of the space defined by the active compounds. The Euclidean space is defined by five simple molecular properties: number of donors, number of acceptors, number of rotatable bonds, XlogP and 0.01 × molecular weight. This concept of ranking compounds by distance in a high dimensional Euclidean property space has recently been published as a virtual screening method, known as DACCS [51].
Fig. 8AUCs for various virtual screening methods on part of the Surflex-Dock validation set
It is clear from Fig. 8 that in four of the eight cases (OPPA, HIV-PR, TK and PARP) the active compounds are very dissimilar from the background set, as the 1D ranking method gives very good virtual screening performance. In a fifth case (TS) the performance of 1D method is as good as any of the 3D methods, although none of the tools perform particularly well. Accordingly, judging virtual screening performance for any tool using such datasets is unlikely to be productive as most of the “signal” separating actives from decoys lies purely in differences in simple molecular properties. A further confounding issue with these datasets is that a large number of the active compounds in these sets are close structural analogues of one another. For ligand-based methods this high structural similarity amongst the actives can cause the actives to be very easily discriminated from the decoys, while a structure-based method may have more difficulty. Accordingly, while property bias is an important consideration in constructing decoy sets, analogue bias should be carefully considered when selecting sets of active compounds.
A recent effort to avoid some pathologies arising from poorly selected decoy sets has come from the work of Huang et al. [52] with the Database of Useful Decoys, or DUD. In this work decoys were selected to match the same simple molecular properties of the active compounds using a similar approach to that mentioned earlier, so that the decoys are not trivially separable from the actives. DUD represents a very wide-ranging dataset (actives against 40 target proteins) that has been designed to evaluate the underlying performance of docking tools, and not the sensitivity of that tool’s scoring metric to differences in simple molecular properties. Note that the DUD decoy selection approach uses a discontinuous representation of the molecular properties, while the approach mentioned above uses a continuous representation. The similarity between these two approaches for decoy selection and the DACCS approach for active selection is striking. It is a topic of further investigation whether much of the reported success of the DACCS method is due to over-training/poor active compound selection.
The work of Huang et al. with the DUD dataset showed that, at least for DOCK, this approach of specifically matching the properties of the decoys to those of the actives can produce more difficult decoy sets than those chosen purely for drug-likeness in a number of cases. It should be noted that the differences in performance reported in the DUD paper are essentially anecdotal, since no error bars are reported. With that caveat in mind inspection of Fig. 4 of the DUD paper, in which comparisons of performance of DOCK between the DUD “own” decoy sets and some commonly available agnostic decoys are shown, is instructive. In 10 of the 12 cases presented the DUD “own” decoy set is the most challenging of the four decoy sets compared, implying that a property-matched decoy set can provide a more difficult background set than an “agnostic” or general decoy set. For a drug-like decoy set chosen without specific reference to the active compounds being screened for, which is therefore to be expected to be less challenging than decoy sets designed with specific reference to the actives being searched for, see Ref [53]. The DUD datasets also have relatively high prevalence (the design goal was to achieve a prevalence close to 3%, though this varies slightly from target to target), giving the results generated with DUD reasonably low errors (except for those cases where the number of actives is small). This is not to say that DUD is perfectly constructed––there are still some large differences between the properties of the actives and the decoys in some cases and in some cases there are so few active molecules that statistically robust results cannot be generated. An important property not considered in the selection of the DUD decoys is formal charge. As such, there are some sets of the DUD actives that are easily discriminated from the decoys based on formal charge. For example, the mean formal charge on the neuraminidase ligands is +1.76, while the mean charge on the decoys is +0.76. For acetylcholinesterase, the mean charge on the actives is −1.68, while the mean charge on the decoys is −0.76. As with the datasets illustrated in Fig. 8, the DUD active sets were not chosen with a view to structural diversity and some of the active sets consist entirely of closely analogous compounds. Very recently the original DUD dataset has been extended by adding more active compounds and by clustering the actives to remove trivially graph similar actives from the set [54]. This makes “DUD 2.0” a suitable dataset not only for docking approaches but also ligand-based techniques. It should be noted that there is a limit to the acceptable level of similarity between actives and presumptive decoys. When the decoys are too similar to the actives the assumption that the decoys are inactive becomes increasingly untenable, giving rise to large numbers of “false false positives”. Accordingly the problem of decoy selection is not yet completely solved, and may not admit of a single solution for all problems or tools. However, since in retrospective work the point is purely to gain a measure of the expectation of performance in as yet unperformed studies, the use of carefully designed decoy sets is mandatory.
It is unfortunate that the docking targets in DUD (39 crystal structures and 1 homology model) were not selected with as much care as the small molecule datasets. In 6 of the 38 co-crystal structures in DUD (there is one apo structure in the set), the DPIs are 1.5 Å or more, resulting in significant uncertainty in the positioning of any atom in these structures. These structures are ALR2 (1AH3), COX-2 (1CX2), EGFR (1M17), GR (1M2XZ), InhA (1P44) and p38 (1KV2). Accordingly docking results from these structures should be interpreted with great care.
Conclusions
Large numbers of evaluations and comparisons of tools for pose prediction and virtual screening have been published in recent years, an indication of significant interest in identifying tools that will have robust performance in one or both of these areas. Unfortunately the vast majority of these studies have been invalidated by poor choice of datasets, lack of consideration of error in source data and use of metrics that do not permit robust comparisons. For those papers using crystal structure data, too little account is taken both of the unavoidable imprecision in these structures and of the errors of fitting that are regrettably frequently seen in structures in the RCSB. In many cases nominal resolution, a measure of the quantity of data gathered, is confused with a measure of quality for the structure and other metrics indicating quality and reliability (DPI and Rfree) are ignored. When performing pose prediction geometric measures such as RMSD are almost always used to compare the experimental and predicted pose. These measures are uniformly used without taking into account either the inevitable imprecision in the atomic positions in crystal structures or the fact that using geometric measures necessarily implies comparing a model for the source data with a computed pose. In almost no cases are crystal structures inspected for errors in fitting. Without taking all these sources of error into account the results of any publication that uses crystal structure data will be suspect and of little use in deciding what tools are the most suitable for the task at hand. In papers concerned with virtual screening there has been, until recently, too little focus on eliminating trivial reasons for good performance from a given tool. The DUD dataset [52] has illustrated ways in which challenging virtual screening datasets can be constructed and, since it is publicly available, DUD offers the opportunity for a common benchmark upon which a wide variety of tools can be compared. The plethora of metrics used to judge and compare virtual screening performance serves merely to confuse the field rather then to clarify it. The lack of confidence intervals on metrics for success makes meaningful comparisons between tools almost impossible to interpret. The AUC for ROC offers great promise as a metric for virtual screening, as it offers the possibility of predictive value along with robust errors. For an exemplary use of ROC in virtual screening tests see a recent paper by Jain [55]. It is hoped that the field will soon converge to a single metric of virtual screening performance, such as the ROC, that will allow robust and direct comparisons between tools and between studies.

pose prediction
virtual screening
coordinate error
property bias
software evaluation


Secondary meningioma in a long-term survivor of atypical teratoid/rhabdoid tumour with a germline INI1 mutation

Objective We report on a patient who developed a meningioma more than two decades after removal at a young age of an atypical teratoid/rhabdoid tumour (AT/RT), which was due to a germline INI1 mutation, and radio- and chemotherapy.
Introduction
We previously reported on a family in which four cousins developed an atypical teratoid/rhabdoid tumour (AT/RT) at young age, due to inheritance of a germline G>A mutation in the donor splice site of exon 4 of the tumour suppressor gene INI1 [2]. More than 20 years after resection of the AT/RT, the oldest of the four cousins developed an intracranial meningioma and, after an additional year, a myoepithelioma of the lip. In addition to the constitutional INI1 mutation and loss of the normal INI1 allele, we detected an identical rearrangement in the NF2 gene, not affecting its coding regions, in the AT/RT and myoepithelioma of the patient and concluded that they both originated from a common precursor cell. However, the rearrangement in NF2 and loss of the normal INI1 allele could not be demonstrated in the meningioma. Despite this, the development of the meningioma may be the consequence of the presence of the constitutional INI1 mutation, because mutations in this gene have been identified in meningiomas [8]. Alternatively, the meningioma could be induced by the radiation therapy, which our patient received after removal of the AT/RT and which is a well-known consequence of this treatment [1]. We investigated the meningioma for the presence of genetic changes reported in radiation-induced meningiomas and performed a comparative loss of heterozygosity (LOH) analysis to demonstrate that such radiation-induced genetic changes are absent in the AT/RT and myoepithelioma of our patient.
Materials and methods
Tumour samples and DNA isolation
DNA from the AT/RT and the myoepithelioma was isolated from formalin-fixed and paraffin-embedded tumour material, and DNA from the meningioma was isolated from a fresh frozen tumour sample using commercially available kits (Qiagen, Venlo, The Netherlands).
SNP analysis
A genome-wide single nucleotide polymorphism (SNP) analysis of normal and meningioma DNA was conducted with the Affymetrix GeneChip Human Mapping 10K Array Xba 142 2.0. The array contains 10,204 SNPs distributed across the genome. The GeneChips were processed by an authorised Affymetrix Service Provider using standard Affymetrix protocols (Service XS, Leiden, The Netherlands). LOH regions were identified by comparing the SNP profiles of normal and meningioma DNA and scoring heterozygous SNP markers (AB) in the normal DNA that were reduced to homozygosity (AA or BB) in the meningioma DNA. Intervals were determined on the basis of locations of SNP markers in the UCSC Genome Browser, March 2006 Build, at http://genome.ucsc.edu/.
LOH analysis
Loss of heterozygosity (LOH) analysis was performed with microsatellite markers as described previously [3] using primer sequences and conditions for polymerase chain reaction given by The Genome Database at http://www.gdb.org/. Because of rather extensive degradation of DNA extracted from the AT/RT, only markers generating PCR fragments smaller than 150 bp were used.
Case report
History
Our patient was diagnosed with a tumour in the fourth ventricle and right cerebellar hemisphere at the age of 4.5 years. Resection of the tumour was incomplete. The tumour was originally classified as anaplastic ependymoma WHO grade III, but upon extensive genetic analysis and histopathological re-evaluation, reclassified as AT/RT [2]. He received adjuvant chemotherapy (methotrexate, vincristine and prednisolone) followed by craniospinal radiotherapy (3,300 cGy in 22 doses) with a boost of 2,100 cGy on the location of the tumour. Fifteen years later, a brain CT without contrast showed no signs of recurrent disease or of other new pathology. Almost 6 years later, 20.5 years after initial diagnosis, he presented again with bifrontal headaches, increasing ataxic gait disturbances and fatigue. CT examination showed a large isodens space-occupying lesion in the left temporal region with oedema, midline shift and contrast enhancement. One year after resection of this tumour, he developed a myoepithelioma of the upper lip. The patient is alive at 29 years of age.
Histological findings
Histopathological examination of the left temporal tumour revealed an atypical meningothelial meningioma WHO grade II with many vessels and up to six mitoses per 2 mm2.
SNP and LOH analysis Radiation-induced meningiomas usually display a complex karyotype [1]. To determine the genetic changes, we performed a genome-wide scan by SNP analysis of the meningioma and corresponding normal DNA. The identified chromosomal regions displaying LOH are listed in Table 1. Next, we tested several markers from within each of the five largest deleted regions by conventional microsatellite analysis to determine their LOH status in the meningioma and in the AT/RT and myoepithelioma. Presence of LOH in the meningioma, but absence of LOH in AT/RT and myoepithelioma was found for D1S199 in 1pter-p34.3 (Fig. 1a), D2S327 in 2pter-p12, D3S3727 in 3p26.1-cen (Fig. 1b), D16S3020 in 16pter-p12.1 and D17S1872 in 17qcen-q21.31 (Fig. 1c). In contrast, as determined previously [2] and exemplified by marker D22S430 in Fig. 1d, there was LOH for chromosome 22 markers in the AT/RT and myoepithelioma, but not in the meningioma.
Fig. 1LOH analysis using markers from chromosome regions a 1pter-p34.3 (D1S199), b 3p26.1-cen (D3S3727), c 17qcen-q21.31 (D17S1872) and d 22q (D22S430) of AT/RT (A), meningioma (M) and myoepithelioma (My) DNAs of the patient. N denotes normal DNA derived from the patient’s blood leucocytes. Arrowhead indicates position of lost alleleTable 1LOH regions in the meningioma, as determined by SNP analysisChromosome regionExact interval (Mb)1pter-p34.30–38.901p33-p3346.99–51.091p32.2-1p31.358.19–65.502pter-p120–79.283p26.1-cen6.11–88.847q36.1-qter150.04–158.6616pter-p12.10–22.7516q23.1-q23.373.46–81.3317cen-q21.3123.09–40.3519pter-p13.20–8.83
Discussion
In a previous study, we provided evidence for the involvement of INI1, by constitutional mutation and independent somatic loss, in the development of both the AT/RT and the myoepithelioma of our patient. Moreover, we could demonstrate that these tumours originated from an identical precursor cell [2]. Although involvement of INI1 has been described in sporadic meningiomas [8], until now meningiomas have not been reported to occur as a consequence of the inheritance of an INI1 mutation. However, this might be due to the extreme rarity of families with an inherited INI1 mutation, with only four published cases so far [2, 5, 6, 11]. In this context, it is interesting to note that (as demonstrated before and exemplified by marker D22S430 in Fig. 1d) the meningioma of our patient has no LOH 22 and the retained INI1 allele carries no additional inactivating mutation [2], indicating that this tumour is not the result of inheritance of the mutated INI1 gene. In contrast, the meningioma displayed genetic changes that are characteristic for radiation-induced meningiomas [1, 7, 9], such as a complex karyotype, as evidenced by the SNP analysis (Table 1), including extensive losses on chromosome arm 1p, but no loss of chromosome 22, and no mutation in the NF2 gene [2]. The genetic changes found in the meningioma proved not to be present in the AT/RT or myoepithelioma (Fig. 1a–c, and data not shown), demonstrating that the meningioma is genetically not related to these tumours and should not be considered as a late and more differentiated recurrence or metastasis of the AT/RT.
In conclusion, the clinical and genetic characteristics of the meningioma clearly indicate that the tumour developed as a consequence of the radiation therapy. Recent evidence suggests that long-term survival of patients with a childhood AT/RT can occur especially after aggressive treatment including radiotherapy [10, 12]. The possible late effects of radiation therapy have been considered as serious drawbacks of this treatment. This is the first case in literature in which such a possible consequence in the form of a radiation-induced meningioma is actually documented in a long-term survivor of an AT/RT. Compared to sporadic meningiomas, radiation-induced meningiomas are more aggressive and have a higher chance to recur [1]. Because of this unfavourable prognosis, continuous medical surveillance of our patient is warranted.

meningioma
atypical teratoid/rhabdoid tumour
ini1
radiation therapy
genetic predisposition


Increased osteoclastic activity in acute Charcot’s osteoarthopathy: the role of receptor activator of nuclear factor-kappaB ligand

Aims/hypothesis Our aims were to compare osteoclastic activity between patients with acute Charcot’s osteoarthropathy and diabetic and healthy controls, and to determine the effect of the receptor activator of nuclear factor-kappaB ligand (RANKL) and its decoy receptor osteoprotegerin (OPG).
Introduction
Although Charcot’s osteoarthropathy is characterised by increased local bone resorption [1], the exact cellular mechanisms contributing to the pathogenesis of this condition remain unresolved. Osteoclasts have been shown to be the principal cell type responsible for bone resorption [2]. These cells originate from the haemopoietic lineage and are known to undergo various stages of proliferation, fusion and differentiation before they are fully functionally active, mature osteoclasts. Recently, receptor activator of nuclear factor-kappaB (RANK) ligand (RANKL) has been identified as an essential mediator of osteoclast formation and activation [3]. RANKL is expressed on a variety of cell types such as bone forming osteoblasts, T lymphocytes, dendritic cells, endothelial cells and fibroblasts. RANKL mediates the process of osteoclastogenesis by binding to its RANK, which is expressed on mononuclear osteoclast precursors. The effects of RANKL–RANK interaction are physiologically counterbalanced by osteoprotegerin (OPG), which acts as a soluble receptor decoy for RANKL and blocks the interaction of RANKL with RANK. The ratio of RANKL to OPG has been suggested to regulate the extent of osteoclast formation and resorption. Therefore, any alteration in the RANKL/OPG ratio could be critical in the pathogenesis of osteolytic bone disorders [4].
Recently, Jeffcoate hypothesised that the RANK/RANKL/OPG pathway may play an important role in the osteolysis seen in acute Charcot’s osteoarthropathy [5]. Using an in vitro technique to generate functional human osteoclasts from peripheral blood monocytes (PBMCs) [6] in the presence of macrophage-colony stimulating factor (M-CSF) [7] and soluble RANKL, it is possible to determine the cellular mechanisms involved in the process of osteoclast formation and resorption in physiological and pathological conditions. To our knowledge, this technique has not yet been studied in patients with Charcot’s osteoarthropathy.
The aims of this study were: (1) to generate functional human osteoclasts in vitro from diabetic patients with acute Charcot’s osteoarthropathy and from healthy and diabetic control participants; (2) to compare the extent of osteoclast formation and resorption; and (3) to determine the role of the RANK/RANKL/OPG pathway in osteoclastic activity in Charcot’s osteoarthropathy.
Methods
Patients
We studied nine consecutive diabetic patients with recent onset of acute Charcot’s osteoarthropathy (five men, four women; five type 1, four type 2 diabetes), eight diabetic patients with no previous history of Charcot’s osteoarthropathy (five men, three women; four type 1, four type 2) and eight healthy control participants (five men, three women). Patients with acute Charcot’s osteoarthropathy were matched for age and duration of diabetes with the diabetic control patients and for age with the healthy control participants. The mean age was similar between patients with Charcot’s osteoarthropathy and diabetic control patients (53 ± 2.8 versus 59 ± 2.9 years [mean ± SEM], p = 0.167) as was the mean age between the former and healthy control participants (53 ± 2.8 versus 47 ± 2.7 years, p = 0.114). The mean duration of diabetes was similar in both groups with diabetes (31 ± 5.1 [Charcot patients] versus 27 ± 4.6 years, p = 0.606). Diabetes control as indicated by glycated Hb was also similar in the two diabetes groups (7.7 ± 0.6 [Charcot’s] versus 7.8 ± 0.4%, p = 0.743).
Diagnosis of Charcot’s osteoarthropathy was made on the presentation of a hot swollen foot, with skin foot temperature 2°C greater than the corresponding site on the contralateral foot and with typical radiological changes of subluxation, dislocation or fragmentation of bone on standard foot radiographs [8]. All patients had intact feet and no evidence of foot infection or ulceration.
Ethical permission for this study was obtained from the King’s College Hospital Research Ethics Committee and all participants gave written informed consent.
Isolation and culture of monocytes
Peripheral blood mononuclear cells were isolated as previously described [6]. Briefly, blood was diluted 1:1 in α-minimum essential medium (MEM; Invitrogen, Paisley, UK), layered over Histopaque and centrifuged (693 g) for 20 min. The interface layer was resuspended in MEM, then centrifuged (600 g) for a further 10 min. The resultant cells were resuspended in MEM with 10% heat-inactivated FCS and counted in a haemocytometer following lysis of erythrocytes by a 5% (vol./vol.) acetic acid solution.
To assess the extent of osteoclast formation and resorption, PBMCs were cultured on glass coverslips and dentine slices. Initially, 5 × 105 PBMCs were added to 6-mm diameter glass coverslips and 4-mm diameter dentine slices in MEM containing 100 UI/ml penicillin, 100 μg/ml streptomycin and 10% FCS (Gibco, Paisley, UK). After 2 h incubation, coverslips and dentine slices were vigorously rinsed in medium to remove non-adherent cells. The cultures were maintained in MEM/FCS under three different culture conditions: (1) human M-CSF (R&D Systems Europe, Abingdon, UK) alone at 25 ng/ml; (2) M-CSF plus 100 ng/ml human soluble RANKL (sRANKL; Peprotech, London, UK) (a concentration known to facilitate differentiation of osteoclast precursors to active bone-resorbing osteoclasts in vitro); and (3) M-CSF plus sRANKL plus 250 ng/ml human OPG (R&D Systems Europe).
Coverslips and dentine slices were cultured at 37°C in 5% CO2 for 14 and 21 days respectively.
Osteoclast formation
After 14 days, the coverslips were examined histochemically for the expression of tartrate-resistant acid phosphatase (TRAcP), an osteoclast marker. Coverslips with newly formed osteoclasts were collected and rinsed in PBS buffer, fixed with formalin (10% [vol./vol.] in PBS buffer) for 10 min and rinsed in distilled water. TRAcP was histochemically revealed by a simultaneous coupling reaction using Naphtol AS-BI-phosphate as substrate and Fast violet B as the diazonium salt. The coverslips were incubated for 90 min at 37°C in a dark room, rinsed three times in distilled water and the residual activity was inhibited by 4% NaF (wt/wt) for 30 min. Coverslips were then rinsed in distilled water, counterstained with DAPI for 20 min and allowed to dry before mounting, using an aqueous medium. TRAcP-positive cells with more than three nuclei were identified as osteoclasts. The number of newly generated osteoclasts was assessed using a light microscope examination.
Osteoclast resorption
After 21 days, the dentine slices were removed from the culture wells, placed in NH4OH (1 mol/l) for 30 min and sonicated for 5 min to remove any adherent cells. They were then rinsed in distilled water and stained with 0.5% (vol./vol.) toluidine blue prior to examination by light microscopy. The surface of each dentine slice was examined for evidence of lacunar resorption and the extent of eroded surface on each dentine slice was determined using image analysis and expressed as the percentage of surface area resorbed.
Statistical analyses
Data were expressed as a mean ± SEM. Initially the difference within the three study groups (Charcot patients, healthy and diabetic controls) was assessed with the non-parametric Kruskall–Wallis test. Then the differences between Charcot and diabetic patients, and Charcot patients and healthy controls were assessed by the non-parametric Mann–Whitney U test. In each patient group, the differences between the various culture conditions were also assessed using the Mann–Whitney U test. Differences were considered significant at p < 0.05.
Results
Osteoclast cultures in the presence of M-CSF
Osteoclast formation The mean number of newly formed TRAcP-positive multinucleated osteoclasts in the presence of M-CSF alone was significantly greater in the patients with acute Charcot’s osteoarthropathy (48.6 ± 18.2) than in diabetic (6.8 ± 2.7) and healthy control participants (5.0 ± 0.7) (p = 0.008). The number of TRAcP-positive multinucleated osteoclasts formed in acute Charcot’s osteoarthropathy was 7.2 and 9.7 times greater than those formed in diabetic (p = 0.010) and healthy control groups (p = 0.003), respectively.
Osteoclast resorption The newly formed osteoclasts exhibited increased functional activity as demonstrated by the extent of resorption on dentine slices, with percentage area resorption significantly elevated in the patients with acute Charcot’s osteoarthropathy (0.264 ± 0.06%) compared with diabetic (0.000 ± 0.00%) and healthy control groups (0.004 ± 0.01) (p < 0.0001). The percentage of resorption was significantly greater in the Charcot patients than in the diabetic (p = 0.001) and healthy control groups (p = 0.001).
Osteoclast cultures in the presence of M-CSF and sRANKL
Osteoclast formation The addition of sRANKL led to an increase in the number of TRAcP-positive multinucleated osteoclasts in all three groups of patients. The mean number of these osteoclasts in patients with acute Charcot’s osteoarthropathy was 96.0 ± 21.6, which was significantly greater than that in the diabetic (56.5 ± 11.5) and healthy (29.0 ± 5.1) control groups (p = 0.010; Fig. 1a,c,e). The number of TRAcP-positive multinucleated osteoclasts in the patients with acute Charcot’s osteoarthropathy was 1.7 times higher than in diabetic control patients, but this finding did not reach significance (p = 0.105). However, the number of these osteoclasts in the acute Charcot group was 3.3 times (and significantly) higher than in the healthy control group (p = 0.005). When the number of cells in the cultures with M-CSF alone was compared with that after the addition of sRANKL, there was a significant increase in the diabetic control patients (from 6.8 ± 2.7 to 56.5 ± 11.5, p = 0.003) and in the healthy participants (from 5.0 ± 0.7 to 29.0 ± 5.1, p = 0.002), while the increase in the number of TRAcP-positive multinucleated osteoclasts in the acute Charcot group failed to reach significance (increase from 48.6 ± 18.2 to 96.0 ± 21.6, p = 0.059; Fig. 2a).
Fig. 1Multinucleated TRAcP-positive cells were formed on glass coverslips (a, c, e) capable of lacunar resorption (b, d, f) after 14 and 21 days incubation, respectively, in the presence of 25 ng/ml human M-CSF and 100 ng/ml sRANKL. Newly formed osteoclasts were numerous and highly active in Charcot’s patients (a, b) compared with diabetic (c, d) and healthy control (e, f) participants. Scale bars, 10 μmFig. 2a Quantitative comparison between the number (n) of TRAcP-positive cells formed in cultures with M-CSF alone (white bars) or with M-CSF and sRANKL (black bars) in patients with Charcot’s osteoarthropathy and diabetic and healthy control participants. b Quantitative comparison between the percentage area resorption in the same cultures and patient groups. Statistical differences between the groups were determined using the Mann–Whitney U test, with significance as follows: a Charcot’s p = 0.059, diabetic control p = 0.003, healthy control p = 0.002; b Charcot’s p < 0.0001, diabetic control p < 0.0001, healthy control p < 0.0001
Osteoclast resorption The percentage area resorption on dentine slices with M-CSF and sRANKL was significantly increased in the acute Charcot group (41.6 ± 8.1%) compared with that in the diabetic (14.2 ± 16.5%) and healthy control groups (10.5 ± 1.9%; p = 0.005). Resorption in the Charcot patients was 2.9 times higher than in diabetic control patients (p = 0.008) and four times higher than in healthy participants (p = 0.005; Fig. 1b,d,f). The addition of sRANKL to the cultures with M-CSF led to the following rises in the percentage area resorption when compared with M-CSF alone: Charcot’s 0.264 ± 0.06% to 41.6 ± 8.1%, p < 0.0001; diabetic control 0.000 ± 0.00% to 14.2 ± 16.5%, p < 0.0001; healthy control 0.004 ± 0.01% to 10.5 ± 1.9%, p < 0.0001 (Fig. 2b).
Osteoclast cultures in the presence of M-CSF, sRANKL and excess concentrations of OPG
Osteoclast formation The addition of excess concentrations of OPG led to a reduction in the number of TRAcP-positive multinucleated osteoclasts in the cultures with M-CSF, sRANKL and OPG in all the three groups of patients. However, after the addition of OPG, the number of TRAcP-positive multinucleated osteoclasts was still significantly increased in the Charcot group (54.4 ± 17.6), as compared with diabetic (8.8 ± 5.3) and healthy control participants (4.4 ± 1.2; p = 0.003). In the cultures with M-CSF, sRANKL and OPG, the number of TRAcP-positive multinucleated osteoclasts was greater in the Charcot patients than in the diabetic (p = 0.005) and healthy control groups (p = 0.001).
When OPG was added to the cultures with M-CSF and sRANKL, the reduction in the number of TRAcP-positive cells in Charcot patients was not significant (96.0 ± 21.6 versus 54.4 ± 17.6, p = 0.189). OPG on the other hand significantly inhibited the number of TRAcP-positive cells in M-CSF and RANKL-mediated cultures from diabetic (reduced from 56.5 ± 11.5 to 8.8 ± 5.3, p = 0.005) and healthy control participants (29.0 ± 5.1 to 4.4 ± 1.2, p = 0.003; Fig. 3a).
Fig. 3a Comparison between the number (n) of TRAcP-positive cells formed in cultures with M-CSF and sRANKL (black bars) or with M-CSF, sRANKL and excess concentrations of OPG (250 ng/ml) (grey bars) in patients with Charcot’s osteoarthropathy and diabetic and healthy control participants. b Comparison between the percentage area resorption in the same cultures and patient groups. Statistical differences between the groups were determined using the Mann–Whitney U test, with significance as follows: a Charcot’s p = 0.189, diabetic control p = 0.005, healthy control p = 0.003; b Charcot's p = 0.001, diabetic control p = 0.001, healthy control p < 0.0001
Osteoclast resorption The addition of OPG led to a marked reduction of the percentage area resorption on dentine slices in Charcot patients (from 41.6 ± 8.1% to 5.9 ± 2.4%, p = 0.001) and also in diabetic (14.2 ± 16.5% to 0.45 ± 0.31%, p = 0.001) and healthy control (from 10.5 ± 1.9% to 0.00 ± 0.00%, p < 0.0001) participants (Fig. 3b).
However, the percentage area resorption on the dentine slices was still greater in the cultures with M-CSF, RANKL and OPG from the patients with acute Charcot’s osteoarthropathy (5.9 ± 2.4%) than in those from diabetic (0.45 ± 0.31%) and healthy control (0.00 ± 0.00%) participants (p = 0.003). Resorption on the dentine slices was greater in the Charcot patients than in diabetic (p = 0.005) and healthy control (p = 0.003) groups.
Discussion
This study shows that monocytes from patients with acute Charcot’s osteoarthropathy cultured in the presence of M-CSF alone were capable of differentiating into mature osteoclasts that exhibited increased resorption compared with diabetic and healthy control participants. Furthermore, osteoclasts generated after the addition of sRANKL were functionally more aggressive, exhibiting a considerable increase in the extent of resorbing activity in patients with acute Charcot’s osteoarthropathy. This resorption was partially blocked by the addition of excess concentrations of OPG, a soluble receptor decoy for RANKL. This suggests that the increased osteoclastic activity in patients with acute Charcot’s osteoarthropathy is mediated through both a RANKL-dependent and a RANKL-independent pathway.
Cultures from the patients with Charcot’s osteoarthropathy showed increased osteoclast formation and resorption when cultured with M-CSF alone. Although M-CSF is an essential factor for proliferation, differentiation and survival of the monocyte-macrophage lineage [9, 10], it is not an osteoclastogenic factor and it is unusual to detect osteoclast formation and resorption in cultures with M-CSF, as was seen in the diabetic and healthy controls. This observation suggests that in acute Charcot’s osteoarthropathy there may be increased levels of other circulating pro-inflammatory factors such as TNF-α [11, 12], IL-6 [13], IL-8 [14] and LIGHT (homologous to lymphotoxins exhibiting inducible expression and competing with herpes simplex virus glycoprotein D for herpes virus entry mediator [HVEM], a receptor expressed by T lymphocytes) [15], which have been previously shown to stimulate osteoclastogenesis independently of RANK/RANKL mechanisms. The concentrations of these circulating factors in diabetic and healthy control participants may not be sufficient to induce the formation and differentiation of active osteoclasts in the presence of M-CSF alone.
After the addition of sRANKL to M-CSF cultures, the newly formed osteoclasts exhibited markedly increased resorption in the patients with Charcot’s osteoarthropathy, although the number of osteoclasts did not significantly increase in these patients compared with cultures with M-CSF alone. These observations may not be unique to Charcot’s osteoarthropathy, and indeed similar observations have been reported in other conditions associated with increased bone resorption, such as rheumatoid arthritis where the addition of sRANKL resulted in a significant increase in lacunar resorption, but did not lead to a significant increase in the number of TRAcP-positive cells [16]. Overall, the observed extensive resorption in acute Charcot patients, in the presence of M-CSF and sRANKL, as compared with the diabetic or healthy control groups, may suggest that the osteoclast precursors circulating in acute Charcot patients are in a higher activated state and as such are more primed to becoming osteoclasts (mediated through RANKL) than those in the control groups.
In order to ascertain that RANKL was a major osteoclastic activator in patients with Charcot’s osteoarthropathy, excess concentrations of OPG, the soluble receptor decoy to RANKL, were added to the cultures with M-CSF and RANKL. The rationale for this approach was that if osteoclastogenesis is mediated solely through RANK–RANKL interaction, addition of excess concentrations of OPG (as had been previously determined to be sufficient to block osteoclastogenesis through RANKL [15]) would completely abolish the process of osteoclast differentiation and activation. In the current study, although osteoclast formation and resorption in the diabetic and healthy control groups was completely blocked by the addition of OPG, the latter did not achieve total inhibition of osteoclast formation and resorption in patients with acute Charcot’s osteoarthropathy. These results suggest that although RANKL-dependent pathways do play a significant role in the osteoclastic activity of Charcot’s osteoarthropathy, an alternative pathway (other than RANK/RANKL) may also be involved. Osteoclastogenic mediators other than RANKL that have been reported to stimulate osteoclast differentiation independently of the RANKL pathway include TNF-α [11, 12], IL-6 [13], IL-8 [14] and LIGHT [15]. In acute Charcot’s osteoarthropathy, it is possible that one or a combination of these factors may have initiated the circulating osteoclast precursors to be in a more ‘primed’ condition, a situation which as such could explain the observed resorption in Charcot monocyte cultures supplemented with M-CSF alone, without the exogenous addition of any osteoclastogenic mediators.
The osteolysis of Charcot’s osteoarthropathy may be explained by our observation that osteoclast precursors from Charcot patients develop into mature osteoclasts that exhibit increased resorptive activity, especially in response to RANKL, unlike the increased resorption in response to bacterial infection, which is not mediated by RANKL [17]. Increased expression of RANKL has been previously demonstrated in pathological osteolysis associated with the development of various bone diseases [18] and a similar mechanism may contribute to osteolysis of Charcot’s osteoarthropathy [5]. Furthermore, patients with Charcot’s osteoarthropathy have severe neuropathy, which itself can also lead to increased expression of RANKL as a result of the loss of nerve-derived peptides known to antagonise its effect such as calcitonin gene-related peptide [5]. In addition to the RANKL-dependent pathway, our results suggest that a RANKL-independent pathway, mediated by pro-inflammatory cytokines, may also be important. Indeed, Charcot’s foot is characterised by excessive inflammation and proinflammatory cytokines have been implicated in its pathogenesis [19]. In support of this, a recent immunohistochemical analysis of bone samples isolated from Charcot’s osteoarthropathy patients showed excessive osteoclastic activity in a microenvironment enriched with mediators of bone resorption (IL-1, IL-6 and TNF-α) [20]. Thus a RANKL-independent pathway, which is also known to play a role in other osteolytic disorders such as rheumatoid arthritis [21] and aseptic loosening [22], could contribute also to the pathogenesis of the Charcot’s osteoarthropathy.
This study has indicated, for the first time that the RANKL-dependent pathway is important in the pathogenesis of Charcot’s osteoarthropathy, thereby raising the possibility of the use of RANKL inhibition in the management of Charcot’s foot. However, our observations also suggest that a RANKL-independent pathway may play a role, but further investigation is required to fully clarify the mechanism involved. If confirmed, specific pharmacological agents that counteract the RANKL-independent pathway, such as anti-TNF strategies, may be useful in the treatment of Charcot’s osteoarthropathy. Whatever the relative importance of either pathway, this in vitro technique of generating human osteoclasts from PBMCs may allow specific characterisation of osteoclastic activity in each patient and could, in the future, lead to individually tailored anti-osteoclastic treatment for the patient with acute Charcot’s osteoarthropathy.

osteoclasts
charcot’s osteoarthropathy
rankl
opg
resorption
osteolysis