Testing new pharmaceutical products in children
Journal: BMJ (2003)

cine or to determine areas of weakness before instruction or practice.The test may also be useful to show competency in evidence based medicine.A passing score could be defined by asking individuals who are agreed to meet or exceed minimum competence to take the test and setting a minimum proficiency score based on the range of these scores. Validity and reliabilityThere are limitations to the validity, reliability, and general utility of the Fresno test.The groups we used to develop and validate the test probably represented the extremes of proficiency, leaving the middle ground relatively under-represented.The properties of the test may change when it used to assess groups of people that are more representative of the full range of proficiency in evidence based medicine.The content of the test is based on the domains of evidence based medicine as promulgated by several widely read authors. 4 9Nonetheless, there may be disagreement about whether these are the most relevant areas or about whether the questions and grading rubric accurately represent ideal content.For example, on the test item about external validity (or relevance) the expert group did not score significantly higher than the novice group.We chose to retain this item because it examines the recently emphasised issue of clinical relevance, 5 6 which we have found useful in our curriculum.As the evidence based medicine evolves, individual items may be more or less representative of current practice.This test relies exclusively on the opinion of experts as the ultimate standard against which candidates are judged.Although expert opinion is the standard when developing tests, practising physicians are more concerned with improved patient outcomes.However, as no test exists that measures patient outcomes, the Fresno test is an improvement over current methods of assessing learning by self report. 1 The inter-rater reliability reported here is high despite the inherent subjectivity of a test of this nature.The two raters participated in the construction and revision of the rubrics and therefore knew them well when scoring these tests.This familiarity with the rubrics may have led to unrealistically consistent scoring.Also, the test presently has only one set of clinical vignettes and one set of numeric examples for calculation questions.We have written, but not tested, new clinical vignettes.Other vignettes will probably be needed if the test is used in other clinical disciplines. ConclusionsThe Fresno test is the first standardised, objective measure of ability in evidence based medicine that requires learners to demonstrate knowledge and skill.It can assess the effectiveness of teaching in evidence based medicine and identify strengths and weaknesses of curriculums and individuals.Further investigation might examine whether reliability and validity extends to new sets of raters and learners in other clinical disciplines and to other clinical vignettes.Medical educators may be further challenged to develop tests that reliably assess use of evidence based medicine in real clinical circumstances, not simulated or prompted by vignettes.We thank John Smucney, Upstate Medical University, New York, for providing the validation data set.