(Too) Great Expectations
Authors: Dan L. Nicolae, Carole Ober
Journal: American Journal of Respiratory and Critical Care Medicine (2009)

The sheer number of genetic association studies of asthma candidate genes has likely resulted in many false positive reports in the published literature. The standard approach to assessing the reliability of reported associations is to replicate the association in an independent sample; the greater the number of “replications” the more confidence one has that the gene is truly involved in asthma susceptibility. On the other hand, lack of replication does not necessarily imply that the original report was a false positive, and for this reason interpreting negative replication studies is more challenging.

In this issue of the Journal, Rogers and colleagues (pp. 1084–1090) investigate whether previously implicated candidate markers and/or genes can be further confirmed using data from a genome-wide association scan (1). The stated objectives of this study were to assess the reproducibility of previously reported associations and to evaluate the contribution of additional genetic variation at these loci. However, this approach can also be used, in principle, to evaluate the power of genome-wide association screening to detect known, or replicated, asthma susceptibility genes and to evaluate the relative contribution of known genes to asthma risk in a single sample. The extent to which this approach can address the uncertainty in the greater than 100 asthma-associated genes that have been reported (2) is less clear.

The investigators evaluated genotype data from the Illumina 550k array in and around 39 asthma candidate genes in 403 white trios (child with asthma and parents) from the Childhood Asthma Management Program (CAMP) study. The genes were selected from literature reviews through July 1, 2008 and met the following criteria: (1) significant association with asthma in at least two populations, (2) at least one significant association study included at least 150 cases and 150 controls or 150 trios, and (3) the published association is with a single nucleotide polymorphism (SNP) and not a microsatellite, SNP, or structural variant. In a two-phase design, the first phase, called “SNP level replication,” examined 93 SNPs that were present on the array and were identical to or had strong linkage disequilibrium (LD) (r2 ≥ 0.8) with a previously reported associated SNP; the second phase, called “gene-level replication,” examined 619 SNPs on the array with minor allele frequencies greater than 0.05 that were in or within 50 kb of the 39 genes. In the first phase a 1-tail statistical test was used, and in the second phase a 2-tailed test was used. The investigators identified modest associations with 10 SNPs in six genes in the first phase (P = 0.01–0.05), and evidence for association with SNPs in 15 genes in the second phase (P = 0.002 to 0.05), although none of the “gene-level” associations were significant after adjusting for multiple comparisons. What do these results tell us about replication of asthma susceptibility genes?

The main challenge in replicating genetic associations is acquiring a sufficiently powered dataset. The power of any one study depends on sample size, the allele frequency of the associated variant, and the effect size of the associated variant or genotype. The first two components are known for a genotyped marker, but the effect size is generally not accurately measured, especially for diseases with strong environmental influences such as asthma.

The effects of sample size and allele frequency on the power of replication are easier to assess if a confidence interval for a measure of effect size, such as the additive model odds ratio (OR), is reported in addition to a P value, which can be misleading. For example, a SNP with an OR of 1.2 in the original (discovery) study reporting the association and a confidence interval of 0.97 to 1.23 in the replication sample has insufficient information for distinguishing between no association (OR = 1) and association. The P value in the replication would not be significant, yet it would be inaccurate to conclude that the study did not replicate the original finding. Moreover, a direct comparison of ORs in the discovery and replication sample allows a less biased evaluation of the evidence for replication compared with just evaluating P values from association tests that only investigate if the ORs are significantly different from 1.0.

The most difficult issue in replication studies and in the evaluation of power is accurately estimating the effect size. Among the factors that are relevant in the consideration of effect size are: (1) we nearly always attempt replication using proxies of causal variants (i.e., using markers in LD with the causal variants) because in the majority of situations the causal/functional SNP is not known; (2) the quality of the proxy can be sample dependent, especially when samples of different race are used (e.g., replication of a SNP associated with asthma in whites using an African American dataset may fail because the proxy SNP may be in LD with the causal SNP in whites but not in African Americans), (3) functional variation may be less well covered than nonfunctional variation on the high-throughput platforms (3), and (4) the effect sizes in the discovery and replication samples are very similar only when then the two cohorts are similar with respect to phenotype (e.g., adult vs. childhood asthma; atopic vs. nonatopic asthma), relevant environmental variables (e.g., exposure to cigarette smoke or farm animals in infancy), and ascertainment (e.g., family-based ascertainment vs. case-control studies; multiplex vs. simplex cases). In fact, there are many examples that demonstrate the importance of considering “environment” in genetic studies. Replications of associations with the CD14 or HLA-G genes were only realized when the samples were stratified by animal/endotoxin exposure (4–7) or maternal affection status (8, 9), respectively. Many other “failed” replications also likely represent our failure to understand the relevant environmental contexts that influence the genotypic effects at the candidate SNP.

Considered in this context, the Rogers and colleagues study tells us little about the 15 genes that were not replicated. On the other hand, this study provides additional evidence for association with 24 asthma candidate genes and is likely the first of many studies that will use genome-wide SNP genotypes for candidate gene studies. As with this study, it will ultimately be much easier to prove a gene's involvement in asthma pathogenesis through replication than to prove that an associated SNP in one study is actually a false positive result.