Metabolife 356, which claims to raise the body's metabolism and help dieters lose weight while maintaining high energy levels, contains 32 ingredients, including ephedra, guarana (an herbal source of caffeine), bee pollen, and caffeine. The product label recommends that adults take one to two caplets two to three times per day or every 4 hours, not to exceed eight caplets per day. Warnings on the product label suggest that a health care professional be consulted by individuals who are using any other dietary supplement, prescription drug, or over-the-counter drug containing ephedrine alkaloids or who have, or have a family history of, any of 11 health conditions, including heart disease, high blood pressure, diabetes, recurrent headaches, and depression. The label also recommends that persons should not use the product for more than 12 weeks and that exceeding the recommended amount may cause serious adverse health effects including heart attack or stroke. Other possible side effects mentioned on the label include rapid heartbeat, dizziness, severe headache, and shortness of breath. The complete product label is in appendix II. The Dietary Supplement Health and Education Act of 1994 created a framework for FDA's regulation of dietary supplements as part of its oversight of food safety. Dietary supplements are generally marketed without prior FDA review of their safety and effectiveness. Manufacturers of dietary supplements are responsible for ensuring the safety of the dietary supplements they sell. Therefore, FDA relies on voluntary reports of adverse events from consumers, health professionals, and others in its effort to oversee the safety of marketed dietary supplements. Although there are no adverse event reporting requirements for manufacturers of dietary supplements, there are such requirements for many other products regulated by FDA. Various types of adverse events associated with the use of human drugs and biologics, animal drugs, animal feeds containing animal drugs, medical devices, infant formulas, and radiation-emitting devices must be reported to FDA. In addition to dietary supplements, other products regulated by FDA that do not require adverse event reporting are foods, cosmetics, and color additives. (See app. III for details about adverse event reporting requirements.) Voluntary adverse event reporting systems can be valuable tools for identifying potentially serious health issues that may be associated with the use of a product and for maintaining ongoing surveillance. FDA has used adverse event reports to identify issues for further investigation and, as we previously reported, it has used adverse event reports to help identify dietary supplements for which evidence of harm existed, and has issued warnings and alerts for dietary supplements. However, by themselves, adverse event reporting systems generally are not sufficient to establish that a product caused the reported health problem. As we noted in 1999, all voluntary surveillance systems, including FDA's adverse event reporting system, have certain weaknesses. These include underreporting, reporting biases, difficulties estimating population exposure, and poor report quality. For example, the Department of Health and Human Services (HHS) Inspector General reported that a study commissioned by FDA estimated that FDA receives reports for less than 1 percent of adverse events associated with dietary supplements. In addition, it is often difficult to rule out other possible explanations for the event; for example, the event may have been caused by preexisting medical conditions, or by the concurrent use of prescription drugs, over- the-counter drugs, or other supplements. For these reasons, data from adverse event reports alone cannot be used to determine if the occurrence of a symptom among product users is unusually high. Between August and December 2002, Metabolife International released copies of 15,948 pages of documents that it said contained call records that reported adverse events associated with Metabolife 356 that the company had received from May 1997 through July 2002. Some pages of call records contained information about more than one call while others did not contain reports of adverse events. Some pages were photocopies or duplicates of other pages. The information about reported adverse events in the 14,684 health-related call records we examined was limited. Most of the call records we reviewed did not completely record demographic or medical history information about the consumer. Information about age, sex, weight, height, the amount of product used, and the duration of use was frequently not recorded. Handwritten call records were difficult to read and interpret. Information was often inconsistent across different versions of the same call record. The call records contained limited information about reported adverse events and consumers. In some cases the evidence for a report of an adverse event was a single health-related word on the call record, such as "seizure" or "stroke." In addition, demographic and medical history information was not consistently recorded in the call records. Most of the call records we reviewed did not record information about the consumers' sex, age, weight, or height. Eighty-eight percent of the call records did not record at least one of these variables. In addition, information about the amount of Metabolife 356 used and the duration of use was not recorded in 27 and 33 percent of the call records, respectively. (See table 1.) The absence of this information makes it difficult to assess whether the call records represent a signal of health concerns related to the consumption of Metabolife 356. Both the amount of product used and duration of use were recorded for 60 percent of the calls reporting adverse events. Relatively few of these records involved consumers who reported taking too much Metabolife 356 or using it for too long a period. Specifically, among call records containing information on the amount of product used or duration of use, 99 and 91 percent of consumers, respectively, reported using the product within the guidelines recommended on the label. The format of the call records varied from brief handwritten notes to typed notations to printed versions of a form used by Metabolife International. In general, less information was recorded for the one-third of call records that were handwritten than all other types of records. For example, calls recorded on a typed form more frequently recorded additional information such as recommendations by Metabolife International to discontinue Metabolife 356 (62 percent) or contact a doctor (54 percent) than did those on handwritten forms (13 percent and 8 percent, respectively). Further, it was often difficult to read handwritten call records. We could not always determine how many calls were reported on a single page since there was rarely a clear delineation of events. Because handwritten call records did not follow a template, we were unable to determine if some information was medical history or symptom information, or if a number was a weight, heart rate, or blood pressure. Information in call records was sometimes inconsistent. Where duplicate call records were available, information about consumers and their usage of Metabolife 356 was sometimes presented differently in the different records of the same consumer call. In addition, Metabolife International officials told us that its nurses sometimes used several different terms to document the same type of adverse event. We found that 14,684 of the Metabolife International call records reported at least one adverse event. Ninety-two of these were for the serious adverse events identified in the proposed label warning for dietary supplements containing ephedra that FDA announced on February 28, 2003. Other adverse events reported included significant elevation of blood pressure, abnormal heart rhythm, loss of consciousness, and systemic rash. We cannot establish that any of the reported adverse events were caused by the use of Metabolife 356. We counted 92 reports of heart attack, seizure, stroke, or death--the serious adverse events identified in FDA's proposed label warning for dietary supplements containing ephedra (see table 2). In its 1997 proposed rule on dietary supplements, FDA also identified other types of adverse events as serious or potentially serious. Table 3 shows our counts for almost all such events. The serious and potentially serious types of adverse events described in FDA's June 4, 1997, proposed rule were reported to the agency prior to June 7, 1996. FDA officials report that some other types of adverse events not included in the table may be considered serious or potentially serious but had not been reported to FDA during the time period considered by its proposed rule. In addition, the 14,684 call records with health-related reports presented a broad range of types of adverse events. Many of the call records contained reports of jitters, insomnia, hair loss, bruising, menstrual irregularities, and sexual dysfunction, as well as vague references to events such as "side effect" or "felt sick." Some reported blood in stool, blood in urine, or blood clots. There were also some reports of visits to emergency departments and hospital admissions. Some call records contained reports of diseases such as pulmonary embolus (a blockage of an artery in the lungs), multiple myeloma, and inflammation of heart tissue. We cannot establish that any of the adverse events reported in the Metabolife International call records were caused by the use of Metabolife 356. As we noted earlier, adverse event reports by themselves are generally not sufficient to establish that a health problem was caused by the use of a particular product. For example, for many adverse event reports it is difficult to rule out other possible explanations for the event-- the event may have been caused by preexisting medical conditions, or by the concurrent use of prescription drugs, over-the-counter drugs, or other dietary supplements. In addition, the limited information available in the Metabolife International call records means that we cannot confirm that a particular adverse event occurred, much less identify a specific cause for it. All the reviews of the Metabolife International call records, including ours, counted reports of serious adverse events. None of the reviews reported identical tabulations of these events. For the set of adverse events that Metabolife International counted--heart attack, stroke, seizure, death, and cardiac arrest--our counts are similar to those of the other reviews (see table 4). In total, we counted 96 such events, Metabolife International counted 78, and the counts of the other reviews ranged from 65 to 107. There are several possible reasons for the slightly different counts of serious adverse events in the different reviews. First, the call records themselves are often difficult to understand and interpret. Second, not all of the reviews included the same set of call records, both because some were completed before all of the Metabolife International call records were released and because the reviews adopted different procedures for identifying and discarding duplicate records. Third, the reviews used different definitions of particular events or established different thresholds for categorizing a particular event. For example, we included reports of "convulsions" in our count of seizures, while some other reviews may not have. Specifically, the counts we report in table 4 for our review and the reviews by Metabolife International and Karch include reports of convulsions, while it is not clear if the other reviewers' counts did. Similarly, we did not count as a report of a heart attack a call record that reported "heart attack?", while at least one other review did. We provided a draft of this report to FDA and Metabolife International for their review. FDA asked us to clarify that it has not conducted its own review of the Metabolife International call records, that we only reviewed reports of adverse events contained in the Metabolife International call records, and that we did not review other reports of adverse events among users of Metabolife 356 that have been received by FDA. In addition, FDA pointed out that, when combined with other information, adverse event reports can help establish that an adverse event was caused by a particular health product. FDA's comments are included as appendix IV. FDA also provided technical comments, which we incorporated as appropriate. In its comments, Metabolife International was primarily concerned about our use of the term "adverse events" to describe the health-related complaints that were reported in the call records we reviewed. We believe that our use of the term is accurate and consistent with its use by FDA and others in the field. Metabolife International also wanted us to clarify that, while it did identify some call records as containing references to types of specific adverse events that have been categorized as serious by others, it has not identified any call records as reporting "serious adverse events." We have made revisions so as not to imply that Metabolife International labeled these events as serious adverse events. Metabolife International also made other comments, which we incorporated as appropriate. As agreed with your office, unless you publicly announce its contents earlier, we plan no further distribution of this report until 30 days from the date of this report. At that time, we will send copies to the Secretary of HHS, the Commissioner of FDA, and others who are interested. We will also provide copies to others upon request. In addition, the report will be available at no charge on GAO's Web site at http://www.gao.gov. If you or your staff have any questions, please contact me at (202) 512- 7119. Another contact and major contributors to this report are listed in appendix V. We reviewed call records and supplementary information voluntarily provided to us by Metabolife International to (1) determine the extent to which information was comprehensive, interpretable, and consistently recorded in the call records, and (2) count the number of call records reporting health-related problems associated with Metabolife 356, and how many of them were serious. During our review we removed duplicate call records and records that did not report health-related events. For each record we recorded demographic information about the individual consumer, other details about the call record and the consumer, and categorized the reported events. From August 2002 through December 2002, Metabolife International voluntarily provided to us 15,948 pages of documentation relating to reports of adverse events among consumers of Metabolife 356. Most of these records were from calls made to the company's consumer health information phone line from May 1997 through July 2, 2002. Other records included e-mail messages and letters that had been sent to the company. Nurses on the staff of Metabolife International documented the calls to the consumer HealthLine in a variety of formats. The records included handwritten notes on a page, typed and handwritten letters, forms with handwritten entries, e-mails, and printed versions of records that had been entered into a database developed by Metabolife International. Many kinds of forms were used to record calls, ranging from simple forms with few spaces or check boxes to full-page forms with multiple boxes for consumer and event-related information. Metabolife International officials told us that health complaints that were noted on product return forms that it received were not in the call records provided to us. Metabolife International also provided to us copies of 46 redacted medical records and a list of corresponding call records. After reviewing these records we found 8 that were not associated with other call records. Five of these records contained enough information to determine the nature of the adverse event and were coded in the same way as other call records. The other medical records were used as additional sources of information for documenting the events and consumer information reported in their corresponding records. While most pages of call records contained information about a single call, some included information about multiple calls on the same page, other calls spanned multiple pages, and some did not include any report of adverse events. Records that spanned multiple pages were often letters to the company, some of which were sent with additional information (such as medical bills). Records that did not report an adverse event were either incomplete printouts of other records from the database, product questions, complaints about not losing weight, or reports of consumer satisfaction. As a result, the number of pages of call records that we received from Metabolife International does not correspond to the number of reports of adverse events. The call records and medical records we received were redacted by Metabolife International to remove personal identifying information such as name, phone number, address, fax number, and e-mail address to protect consumer privacy. Metabolife International officials told us that in the process of redacting the records, some relevant adverse event information was also inadvertently removed. Metabolife International officials told us that there were duplicate call records in the set of call records they provided to us. Some duplicate reports were photocopies of the same call record. In other cases, there were multiple versions of the same call record in different formats. Metabolife International officials reported these multiple versions were the result of nurses taking handwritten notes and later entering the same information directly into a database established in September 1999. Metabolife International gave us lists of those call records it believed to be duplicates. Over the course of our review, it identified more than 2,200 records for which there were at least one duplicate. Metabolife International officials reported that they identified the duplicates on the basis of the name of the consumer. Duplicates may have included subsequent calls about different events from the same individual. We examined the duplicate call records identified in the lists provided throughout our review by Metabolife International. Because identifying information was removed, we examined the date of the call record, demographic information about the consumer (such as age, height, weight, the amount of the product used, and duration of use), and event details to determine if they were duplicate records. Where this information was the same or similar, we considered the records to be duplicates and excluded the extra records from our review. We did, however, include in our analysis any additional information that appeared on the duplicate records. For example, if one version included height and another weight, we recorded both of these. We agreed with Metabolife International that most of the more than 2,200 records it identified as duplicates were, in fact, duplicates. However, we did not exclude records that represented multiple calls from the same consumer for different events if the dates on the call records differed by more than a few days or the symptoms were clearly different. During the course of our review, we also identified duplicates not previously identified by Metabolife International, including photocopied records and records that used identical language in event descriptions. We do not know if all duplicate call records were identified. We also excluded from our analysis records in which there was no health complaint or the health complaint could not be clearly determined. We also excluded call records that reported third-hand knowledge of adverse events (such as a friend of a friend who experienced an adverse event). In addition, we did not count call records that clearly referred to nutrition bars or other ephedra-free products manufactured by Metabolife International. In total, we determined that the 15,948 pages of documentation provided by Metabolife International contained 14,684 separate health-related call records. We classified the adverse events reported in each call record and entered the appropriate codes into a database. We classified the reported adverse events as either one of the events FDA identified as serious in its February 28, 2003, announcement regarding a proposed label warning for dietary supplements containing ephedra (heart attack, stroke, seizure, or death) or as an other adverse event. All serious events reported within a particular call record were counted. Therefore an individual could have reported multiple serious adverse events, though this happened in few records. For other adverse events, we documented whether the call record reported one or more adverse events. We did not count the number of reports for every type of event reported in the record. We did, however, count the number of all but 1 of the 24 other types of adverse events that were described as serious or potentially serious in FDA's June 4, 1997, proposed rule on dietary supplements containing ephedrine alkaloids. The set of events identified by FDA in the proposed rule is not an exhaustive list of the adverse events that may be associated with the use of dietary supplements containing ephedrine alkaloids. FDA officials told us that some other types of adverse events may be considered serious or potentially serious but had not yet been reported to FDA during the time period considered by its proposed rule. We did not apply medical judgment in the process of identifying and classifying events. Our classification of events in the call records was based solely on the words and phrases therein; we did not diagnose a consumer's condition or otherwise interpret the information presented. For example, if a report said "poss. heart attack," "heart attack symptoms," or "heart attack?", we did not classify it as a heart attack since it was not clear that a heart attack was reported. Also, while we counted "blood pressure 210/120" as an instance of significantly elevated blood pressure because it reported measurements greater than 160 systolic or 100 diastolic, we did not place in the same category call records that reported only "high blood pressure" because they did not contain the specific measurements needed for that determination. We used MEDLINE Plus Medical Encyclopedia definitions to further clarify individual symptoms related to these categories. We also did not attempt to determine whether Metabolife 356 caused the reported adverse events. Adverse events about many types of products regulated by FDA are required to be reported to the agency. Such products include human drugs, biologics, animal drugs, animal feeds containing animal drugs, medical devices, infant formulas, and radiation-emitting devices. There are, however, no reporting requirements for adverse events associated with other products regulated by FDA, including food and food additives, dietary supplements, cosmetics, or color additives. (See table 5.) Carolyn Feis Korman, Chad Davenport, Julian Klazkin, and Roseanne Price also made major contributions to this report.

Summary:
Dietary supplements containing ephedra, such as Metabolife 356, have been associated with serious adverse health-related events. In a February 28, 2003, announcement, the Food and Drug Administration (FDA) proposed that dietary supplements containing ephedra include a statement on their label warning that "Heart attack, stroke, seizure, and death have been reported after consumption of ephedrine alkaloids." GAO was asked to review health-related call records that Metabolife International--the manufacturer of Metabolife 356--collected from consumers from May 1997 through July 2002. Most of the records were from calls to a consumer phone line the company maintained. Metabolife International voluntarily provided the call records to GAO. Specifically, GAO (1) examined the extent to which consumer information in the call records was comprehensive, interpretable, and consistently recorded, (2) counted the number of call records reporting types of adverse events that FDA had identified in 1997 as serious or potentially serious, and (3) compared GAO's findings to those of six other reviews of the call records, including one by Metabolife International. Adverse event reports generally are not sufficient on their own to establish that reported problems are caused by the use of a particular product, but can signal potential health problems that deserve investigation. The information in the Metabolife International call records was limited. Call records were sometimes difficult to understand, and consumer information was not consistently recorded. In some cases, the evidence for a report of an adverse event was limited to a single word on the record. Most call records also did not record complete information about potentially relevant items such as the consumer's age, sex, weight, and height. Information about both the amount of product used and the duration of use was recorded for 60 percent of the call records. Handwritten call records were difficult to read and understand. By GAO's categorization, 14,684 of the call records contained reports of at least one adverse event. GAO found that there were 92 reports of the serious adverse events identified in FDA's proposed label warning--18 reported heart attacks, 26 reported strokes, 43 reported seizures, and 5 reported deaths. Other types of adverse events identified as serious or potentially serious by FDA in 1997 that were reported in the call records included significant elevation in blood pressure, abnormal heart rhythm, loss of consciousness, and systemic rash. Because of the inherent limitations of adverse event reports and the incomplete nature of these call records, it can not be established from the information available to GAO that the adverse events reported were caused by Metabolife 356. All of the reviews of Metabolife International call records--one by Metabolife International; three by consultants commissioned by Metabolife International; one by the minority staff of the Committee on Government Reform, House of Representatives; one by the RAND Corporation; and one by GAO--found reports of serious adverse events, although none reported identical results. For the set of adverse events counted by Metabolife International--heart attack, stroke, seizure, death, and cardiac arrest--GAO's counts were similar to those of the other reviews. GAO counted 96 such reports and the counts of the other reviews ranged from 65 to 107. In commenting on a draft of this report, FDA discussed the value of reports of adverse events in helping to understand the causes of such events.