Evidence‐based periodontology, systematic reviews and research quality
Authors: Ian Needleman, David R. Moles, Helen Worthington
Journal: Periodontology 2000 (2005)

Periodontology has a rich background of research and scholarship. A simple MEDLINE search of ‘Periodontal Diseases’ OR ‘Periodontitis’ alone from 1966 to 2003 brings up more than 45,000 hits. Therefore, efficient use of this wealth of research data needs to be a part of periodontal practice. Evidence-based periodontology aims to facilitate such an approach, accelerating the introduction of the best research into patient care. This chapter will review the concepts of evidence-based periodontology, introduce the systematic review as a research tool and examine how evidence-based periodontology can both inform on and benefit healthcare in periodontology. Finally, we will examine the strengths and limitations of different research designs and their appraisal. We hope that the information in this chapter will provide a basic understanding of the concepts that will be relevant to reading and enjoying the other chapters in this volume of Periodontology 2000. Evidence-based periodontology is the application of evidence-based health care to periodontology. A useful definition of evidence-based health care has been proposed by Muir Gray: ‘An approach to decision making in which the clinician uses the best evidence available, in consultation with the patient, to decide upon the option which suits that patient best’ (8). Therefore, evidence-based periodontology is a tool to support decision making and integrating the best evidence available with clinical practice. The highest quality evidence will be used if it exists, but if it does not, lower levels of evidence will be considered. Lower levels of evidence usually means research designs more prone to bias and therefore with less reliable data. However, the nature, strengths and weaknesses of the evidence will be made clear to the reader. In addition, wherever possible, the data presentation supplies more clinically relevant information, including the probability of achieving a certain effect such as a benefit, and considering possible adverse effects. Evidence-based periodontology is not simply systematic reviews of randomized controlled trials, although this can be an important aspect. Evidence-based periodontology is an approach to patient-care and nothing more. The expectations that are sometimes laid on it can be inappropriate. It cannot provide answers if research data do not exist (other than using expert opinion) and it cannot substitute for highly developed clinical skills. Therefore, it can never be cookbook healthcare or use statistics in isolation to drive clinical care. Instead it is the comprehensive integration of appropriate research evidence, patient preference and clinical expertise (Fig. 1). How evidence-based periodontology fits into healthcare. Reproduced with permission from Clarkson, J, Harrison, JE, Ismail, AI, Needleman, IG, Worthington, H, eds. Evidence Based Dentistry for Effective Practice. London: Martin Dunitz, 2003 (20). This can be illustrated with data from a recent systematic review on periodontal plastic surgery for root surface coverage in localized Miller Class I and II defects (25). The data from the systematic review demonstrated that connective tissue grafts were significantly better than guided tissue regeneration in reducing recession (mean difference 0.43 mm, 95%CI [0.62,0.23], chi square for heterogeneity 7.8 (df = 5) P = 0.17). This indicates that the pooled difference between six studies included in the review is 0.43 mm, with a 95% confidence interval from 0.62 to 0.23. The chi-square test indicates that there is no evidence of any heterogeneity between the studies (they could theoretically all be measuring the same difference). So, does this mean that only connective tissue grafts should be used in the treatment of localized recession defects? Clearly, this would not be appropriate. The data show that both GTR and connective tissue grafts can work. For the selected outcome, which was recession reduction, connective tissue grafts produce 0.43 mm greater effect; the result is both reasonably precise (judged by the confidence interval) and the studies from which the data were taken were similar (no evidence of heterogeneity). However, recession reduction might not be the only outcome of interest. The two surgical procedures are very different. One requires the harvesting of a soft tissue graft from the palate and the other does not. There are no data available examining patient preferences, but it is likely that some individuals will prefer a procedure that does not involve two surgical sites, even if it does not reduce recession to the same extent. It is also possible that aesthetics are different following the two procedures and this might inform on the decision. However, surprisingly, no data are available on patient views on aesthetics comparing the two procedures. Therefore, this evidence-based approach to management of recession has produced the best available evidence, shown how precise this estimate actually is, and highlighted the limitations of the evidence, in this case the lack of data on some outcomes that are relevant to the decision making process. One of the barriers to the application of research findings in clinical practice is the way that results are often presented. Typically, a mean value will be published, based on a statistical analysis comparing experimental groups. Such a value in conjunction with its associated 95% confidence interval is useful to determine whether there is a statistically significant difference between groups and will often be a requirement of a study designed for regulatory approval. However, this type of analysis is not designed to provide information about the probability of achieving a certain outcome were the reader to apply it in practice. Such an outcome could include achieving a health benefit or preventing further disease. For instance, in a meta-analysis from a systematic review on guided tissue regeneration (GTR) for periodontal infrabony defects, the additional benefit of using GTR over access flap surgery was a 1.1 mm gain in clinical attachment (21, 22). This should, however, not be interpreted as the additional benefit to be expected every time that GTR is used instead of access flap surgery. One approach to analysing and presenting data in a more clinically useful format is to calculate the number needed to treat (NNT). This is the number of patients that would need to be treated to achieve a stated benefit (NNTb) or to avoid a stated harm (NNTh). It is derived from a dichotomous outcome such as the proportion of sites achieving at least 2 mm gain in attachment. For the GTR meta-analysis, and using this benefit, the NNTb is eight. In other words, for every eight patients treated with GTR, you can expect one to have at least 2 mm more gain in clinical attachment than if you had used an access flap (95% confidence interval [4,33]). For detailed guidance regarding the use and calculation of the NNT the reader is recommended to the electronic journal ‘Bandolier’:http://www.jr2.ox.ac.uk/bandolier/booth/painpag/NNTstuff/numeric.htm. High quality research and the use of evidence are fundamental to both evidence-based periodontology and traditional periodontology. The differences between these approaches emanate from how research informs clinical practice. Evidence-based periodontology uses a more transparent approach to acknowledge both the strengths and the limitations of the evidence. An appreciation of the level of uncertainty or imprecision of the data is essential in order to offer choices to the patient regarding treatment options. Evidence-based periodontology also attempts to gather all available data and to minimize bias in summarizing the data. These aspects are key to decision making and are highlighted in Table 1. Furthermore, evidence-based periodontology acknowledges explicitly the type or level of research on which conclusions are drawn. The research hierarchy is discussed in more detail later in this chapter. However, one aspect that influences the reliability of the data is the control of bias. Bias is a collective term for factors that systematically distort the results of research away from the truth. Different research designs offer different possibilities for the control of bias and therefore vary in their reliability and will be discussed further below. An overview of the components is given in Fig. 2. Evidence-based periodontology starts with the recognition of a knowledge gap. From the knowledge gap comes a focussed question that leads on to a search for relevant information. Once the relevant information is located, the validity of the research needs to be considered in two broad areas. Firstly, is the science good (internal validity)? Internal validity focuses on the methodology of research. Secondly, can the findings be generalized outside of the study (external validity)? External validity might be affected by the way treatment was performed. For instance, if the time spent on treatment was extensive it might not be practical to provide this therapy outside of a research study. Another example could relate to the use of many specific inclusion criteria in a trial which could make it difficult to generalize the findings to a wider group of patients. The question the reader should ask is whether their types of patients are so different from the study that it is reasonable to expect differences in outcomes. After locating and appraising the research, the results then need to be applied clinically, or at least included in a range of options. Finally, the results in clinical practice need to be evaluated to reveal whether the adopted technique achieved the expected outcome. The steps of evidence-based periodontology. Reproduced with permission from Clarkson, J, Harrison, JE, Ismail, AI, Needleman, IG, Worthington, H, eds. Evidence Based Dentistry for Effective Practice. London: Martin Dunitz, 2003 (20). The example of gingival recession mentioned earlier can be used to illustrate this approach. The uncertainty might relate to whether to change from using connective tissue grafts for recession defects to guided tissue regeneration and can be translated into a focussed question. Here the patient or problem group could be refined more closely to localized recession defects and perhaps Miller Class I or II, as we might reasonably expect these lesions to respond differently from more advanced lesions. The intervention is guided tissue regeneration and the comparison, connective tissue grafts. The outcomes would include change in recession or possibly the chance of achieving complete root coverage. Since the procedure is primarily for aesthetics, a patient-centred assessment of aesthetics should be an outcome. As always, there must be a consideration of adverse effects and these might include pain, postoperative infection, and severe bleeding postoperatively. Reassembling this structure into a focussed question would lead to ‘In patients with localized Miller Class I or II recession defects, what is the effect of guided tissue regeneration vs. connective tissue grafts on change in recession, chance of complete defect coverage and aesthetics and what are the adverse effects?’ For this particular research question, the randomized controlled trial is best able to address the change in recession outcome. For the other outcomes, other research designs might have been used, such as observational studies. Preferably, we would like to find a systematic review that will have completed the searching and study appraisal for us. The search quickly identifies a systematic review (25). The review has a research question that is appropriate to our question and demonstrates a statistically superior effect of connective tissue grafts compared with guided tissue regeneration. The review also acknowledges certain limitations. In terms of the validity of the meta-analysis, the reviewers urge caution as publication bias could be affecting the overall result, but this could not be tested due to the low number of studies. Publication bias is discussed later in this chapter. Another limitation was that there were no data on aesthetics or adverse effects. Therefore, having reviewed the data, it is clear that there is good evidence to indicate that connective tissue grafts have a greater effect on change in recession than guided tissue regeneration, although there are several limitations to this evidence. Clinical recommendation is tempered by the lack of data on aesthetics and adverse effects and the possible exaggeration of benefit through publication bias. This information can then inform on the case presentation to the patient and a choice of options discussed and agreed. The outcome of treatment can then be evaluated to see whether the desired endpoint was achieved and this helps to refine the case presentation discussion in future. One important element of evidence-based periodontology is the systematic review. Systematic reviews are a research design termed ‘research synthesis’. That is, they use research methodology to pool data from multiple studies that address a particular hypothesis. A systematic review can be defined as a review of a clearly formulated question that attempts to minimize bias using systematic and explicit methods to identify, select, critically appraise and summarize relevant research. The description of systematic reviews as providing the highest level of evidence is widespread but also raises expectations that may or may not be fulfilled. A realistic understanding of what a systematic review can provide is important for the appropriate use of this type of evidence (Table 2). More detailed information on systematic reviews exist (5, 19), and guides to conducting them are freely available (1, 13). As with all research, a systematic review starts from an hypothesis. This is derived from a focussed question which is set to answer a particular area of uncertainty. For instance, for the systematic review on smoking and periodontal therapy in the chapter by Labriola et al. in this volume, the focussed question was: “In patients with chronic periodontitis, what is the effect of smoking or smoking cessation on the response to nonsurgical periodontal therapy in terms of clinical and patient-centred outcomes?”(14). The question has set the types of patients (individuals with chronic periodontitis undergoing nonsurgical therapy), type of exposure (cigarette smoking) and types of outcomes (clinical and patient-centred) to be investigated, each aspect being defined in more detail within the protocol. As this is a prognostic research question, where exposure (smoking) cannot be randomized, the cohort study is the research design of choice to incorporate into this investigation. These components help in the design of the search strategy that aims to be comprehensive. Usually, searching of multiple electronic databases is carried out together with searching other sources. The most commonly searched databases include MEDLINE (strong on English-language studies), EMBASE (strong on other European languages), and CENTRAL (the Cochrane Collaboration register of trials records). Searching only electronic databases can miss important data, as records on the database may not be appropriately coded. To supplement the electronic search, other approaches are used. Typically, this will include checking for publications in the bibliographies of retrieved studies and review articles, hand-searching of journals for missed reports, and contacting researchers, industry and journals for unpublished data. The search strategy aims for high sensitivity, i.e. the greatest chance of finding all relevant studies. The downside of this approach is low precision, i.e. in addition to the relevant studies, the search will identify many irrelevant hits (probably more than 90% of hits from the search will not be relevant). For example, in a systematic review on systemic antimicrobials, the search identified 1,300 hits. Screening of the title and abstracts (if available) indicated that 158 papers might be relevant. Once the full text of the studies had been reviewed, 25 trials were judged relevant and could be included (9). At first sight, rejecting 1,275/1,300 studies would appear to be wasting potentially useful data. However, the deliberately inclusive search identifies a large number of irrelevant papers, including veterinary medicine, review papers, duplicate reporting of research and laboratory studies. The systematic review screens the search findings against prestated criteria. These criteria aim to exclude studies irrelevant to answering the question, but do not attempt to exclude on the basis of the quality of the study. Instead, the quality of relevant studies is critically appraised using objective criteria that could influence the study outcome. The dimension of quality can be incorporated into a systematic review in a number of ways. If the studies are similar enough to be combined in a meta-analysis, the impact of quality on the overall result can be estimated (through sensitivity analyses or meta-regression). If meta-analysis is not possible, the quality of studies can be summarized in narrative tables, in particular for those elements designed to protect against bias. Whilst this may not be as powerful as the use of meta-analysis, it will highlight limitations to be placed upon the conclusions. Following pooling of the data with meta-analysis or qualitative methods, the conclusions from the investigation can be drawn and related to the data derived from the review. The systematic review will not be appropriate for some questions. For instance, to address the question, ‘Which indices have been used to measure gingivitis?’ a descriptive survey will be more appropriate. However, systematic methods should be adopted for some aspects, in particular to ensure that the search is both comprehensive and contemporary. This might form an important initial stage to answering a question such as ‘Which gingivitis indices have been validated?’ This is a research question answerable by a systematic review. Evidence-based periodontology is built upon developments in clinical research design throughout the 18th, 19th and 20th centuries (15, 20, 23, 28). Evidence-based medicine has only been known for just over a decade and the term was coined by the clinical epidemiology group at McMaster University in Canada (4). The influence of the McMaster group spread far. One of the earliest to take up the challenge in periodontology (in fact in oral health research overall) was Alexia Antczak Bouckoms in Boston, USA. Antczak Bouckoms and colleagues challenged the methods and quality of periodontal clinical research in the mid 1980s (3) and set up an Oral Health Group as part of the Cochrane Collaboration in 1994. The editorial base of the Oral Health group subsequently moved to Manchester University in 1997 with Bill Shaw and Helen Worthington as co-ordinating editors (http://www.cochrane-oral.man.ac.uk/). The first Cochrane systematic review in periodontology was published in 2001 and researched the effect of guided tissue regeneration for infrabony defects (21). Many individuals have been active in the critical analysis of the periodontal literature. These include Jan Egelberg, Loma Linda University, Noel Claffey, Trinity College Dublin, and Gary Greenstein, University of Medicine and Dentistry of New Jersey. There have been many notable events in evidence-based periodontology. The 1996 World Workshop in Periodontology held by the American Academy of Periodontology included elements of evidence-based healthcare, supported by Michael Newman at UCLA (2). The 2002 European Workshop on Periodontology became the first international workshop to use rigorous systematic reviews to inform the consensus. The workshop was organized by the European Academy of Periodontology for the European Federation of Periodontology, under the chairmanship of Professor Klaus Lang. Sixteen focussed and rigorous systematic reviews formed the basis of intense consensus discussions. A similar approach was used subsequently by the American Academy of Periodontology for the Contemporary Science Workshop in 2003. Many other groups are now using similar methods in healthcare and research. Most recently, the International Center for Evidence-Based Oral Health was launched in 2003 (http://www.eastman.ucl.ac.uk/iceboh) to produce high quality evidence-based research with an emphasis on, but not limited to, periodontology and implants and to provide generic training in systematic reviews and research methods. Different clinical research questions require evaluation through different study designs. A study to determine the effectiveness of surgical therapy compared with nonsurgical debridement deals with the effectiveness of a treatment option and would be best answered by a randomized controlled trial (RCT) or, ideally, a systematic review of RCTs. However, it must be noted that although RCTs and systematic reviews of RCTs may well be the ‘gold standard’ upon which to base decisions on the effectiveness of interventions, they are not necessarily appropriate, or ethical, to answer all questions. An RCT would obviously not be helpful in answering the question posed on the epidemiological evidence of plaque in the etiology of periodontitis. For such questions regarding prognosis or etiology, cohort studies would be more appropriate. Table 3 illustrates the types of study designs most suitable for different types of research questions arising in periodontology. The most appropriate source of information will depend upon the type of study design being sought. Evidence-based periodontology, as its name implies, is periodontology that is based on evidence, but not just any so-called evidence. Richards wrote a toolbox article for the journal Evidence-based Dentistry entitled ‘Not all evidence is created equal’ (24). We have already seen in this chapter that the quality of evidence may vary according to study design and that this has led to the concept that there can be a hierarchy of evidence. One hierarchy is illustrated in Table 4 and is specific to studies on therapy, prevention, etiology, and harm. Other suggested levels for different types of research question can be found at the Center for Evidence-Based Medicine: (http://www.cebm.net/levels_of_evidence.asp#levels). The publication of research in a high-ranking journal may not be an absolute guarantee of quality. Within the medical literature there are methodological studies which have empirically shown that quality is not merely a hypothetical concept but also affects study outcomes. As examples of this, the reviews of Schulz et al. (26), Moher et al. (17) and Juni et al. (12) showed that in studies in which there was inadequate concealment of treatment allocation, the treatment effects were exaggerated by about 40% compared to trials of higher quality. Two recent studies have investigated the methodological quality of RCTs in periodontology and implantology as assessed by their publications. Both studies targeted RCTs for investigation due to the importance of the RCT in providing evidence for the effect of interventions and also because of the empiric data indicating the effect of key domains of methodology on bias. Montenegro et al. (18) conducted a systematic review of the quality of RCTs of periodontal therapy, published in Journal of Periodontology, Journal of Clinical Periodontology or Journal of Periodontal research over a 3-year period from 1996 to 1998. From the electronic search, 283 papers were possibly relevant and 177 studies met the inclusion criteria of being an RCT, performed on humans and for which a full text article was available. Screening and data abstraction were performed independently and in duplicate to minimise error and bias. The evaluation was not performed blind to author affiliation identity of the RCTs as the evidence suggests that this has a minimal impact on outcome (16). In view of the empirical data described above, the quality components chosen were those demonstrated to be important for protection from bias: adequacy of method of generation of the random sequence, adequacy of method of concealing the allocation sequence from the patient recruitment, examiner blinding (where it was judged possible to achieve), and handling of losses and withdrawals. The results indicated that 29/177 (17%) of RCTs employed a clearly adequate method of generating the random number sequence, and that 12/177 (7%) of studies described adequate allocation concealment (Fig. 3). Furthermore, where examiner blinding was possible, 97/177 (55%) of studies reported an adequate method. Clear accounting for study subjects was present in 100/177 (55%) of reports. Since the study was conducted on trial reports, it is not clear how much of the inadequacy was due to incomplete reporting rather than inadequate study methods. If the data do reflect study conduct, then bias and exaggeration of the effect of the test interventions could be a problem with some trials in periodontology. Quality of reporting of randomized controlled trials in periodontology (18). Percentage of studies with adequate method. Similar results were found when investigating RCTs of oral implants (6). This study searched for RCTs up to the end of 1999 in multiple databases. Seventy-four publications were located and 43 RCTs were quality assessed as many studies were presented in multiple publications. Although the methods and criteria were a little different for this study compared with the quality appraisal of periodontal studies, the results are broadly comparable. A clearly adequate method of randomization/concealment of allocation was present in 1/43 (2%) papers. Blinding was described in 12/43 (28%) studies and the reasons for withdrawals and losses to follow-up were specified in 33/43 (77%) of reports. The adequacy of reporting of clinical research is crucial if the reader is to evaluate the quality and possible impact of studies. The importance of several of the quality issues that we have described has not been thoroughly appreciated until relatively recently. Therefore, it is unfair to judge the past from the standpoint of current knowledge. In addition, the pressure on page numbers in paper-based journals can restrict detail. Hopefully this aspect will be alleviated by initiatives in electronic publication. Guidelines are available to help the publication of clinical research. These guidelines are well accepted by high impact biomedical journals and offer guidance not only to authors but also to editors and reviewers. These guidelines include CONSORT (Consolidated Standards of Reporting Trials) for reporting randomized controlled trials and STARD (Standards for Reporting of Diagnostic Accuracy) for reporting studies on diagnostic tests (http://consort-statement.org/). In addition, three guidelines for reporting systematic reviews are available: QUOROM (Quality of Reporting of Meta-analyses) (http://consort-statement.org/), MOOSE (Meta-analysis Of Observational Studies in Epidemiology) (27), and QUADAS (Quality Assessment of studies of Diagnostic Accuracy included in Systematic reviews) (29). For clarification, it should be remembered that systematic reviews are termed meta-analyses by some in North America, whereas the term meta-analysis is usually reserved only for the statistical combining of data which may or may not be part of a systematic review. The format of these guidelines is similar. Each presents a checklist of items for incorporation into the research report. The selection of items is evidence-based as far as possible and otherwise derived by a Delphi approach to consensus. In addition to the checklist, a chart is used to illustrate the flow of patients through the study. The checklist and chart for CONSORT are illustrated in Fig. 4a,b. The checklist should accompany the manuscript in its journal submission but not be part of the final paper. The intention with the chart, however, is that it should be published as part of the paper. Whilst the checklist has numerous items, each can be concisely addressed and is unlikely to be the main cause for excessive length of a publication. a ) CONSORT Checklist of items to include when reporting a randomized trial. b) CONSORT Flow chart. Available from: http://www.consort-statement.org/. At the time of writing, oral health journals that have adopted CONSORT as editorial policy and their dates of adoption are: British Dental Journal (1999), Journal of Orthodontics (2000), International Journal of Endodontics (2003) and Journal of Dental Research (2004). The British Dental Journal is the only one that has adopted QUOROM (2002). Given that some evidence is ‘better’ than other evidence, it seems reasonable to place greater emphasis on ‘good’ than on ‘poor’ quality evidence when making clinical decisions. The problem arises as to how exactly we decide what constitutes good quality evidence. This process is critical appraisal. The validity of published evidence is potentially affected by the quality of every stage of the experimental process from aims and objectives, through design, execution, analysis, interpretation, and finally publication. Although deliberate deception is always a possibility, the majority of problems that arise are in fact unintentional. Most methodological errors may be classified as being the result of bias, confounding, or chance. Therefore, for the purpose of this chapter, quality will be discussed in relation to these methodological issues. Other aspects of study conduct may well be critical to the validity of a study but will not be considered in this chapter as they will be specific for a particular study. Such factors could include how well treatment or supportive maintenance was provided.