CMS administers the 1-800-MEDICARE help line to answer beneficiaries’ questions about Medicare eligibility, enrollment, and benefits. The help line currently operates 24 hours a day, 7 days a week, and has eight call center locations that are run by two contractors. As of October 2004, the primary contractor managed 1,332 of the 2,137 CSRs and operated seven of the eight 1-800-MEDICARE call centers. In addition, the primary contractor is responsible for other activities, such as distributing program material requested by callers, training all new CSRs before they handle calls on the 1-800-MEDICARE help line, and researching answers to more complex questions some callers may have. Prior to 2004, one contractor managed the 1-800-MEDICARE help line. In June 2004, in response to increasing call volume, CMS hired a second contractor, which in October 2004 managed 805 CSRs and operated one of the eight 1-800-MEDICARE call centers. A call placed to 1-800-MEDICARE is answered initially by an interactive voice response system. This is an automated system that, depending on the caller’s responses to the system’s automated prompts, routes a call to a CSR or to other information sources. These other information sources can include the other help lines maintained by Medicare’s claims administration contractors or recorded information. All CSRs must have a high school diploma or its equivalent, but they are not required to be knowledgeable about the Medicare program at the time they are hired. To help provide clear, accurate, and timely answers to callers’ questions, CMS expects the CSRs to use written scripts, which contain information about the program. CSRs listen to a caller’s question and then type in related keywords to generate a list of suggested scripts that could be used to answer the question. The CSRs select the script they consider best suited to answer the question and read either excerpts or the entire script. CSRs can also consult other information sources. For example, CSRs can use Web-based tools available on the Medicare Web site to help beneficiaries select a prescription drug discount plan, nursing home, or home health agency. Because the types of questions frequently posed to 1-800-MEDICARE change in response to program or other policy changes, new scripts may need to be created or existing ones updated. Either CMS or the primary contractor may decide to develop a new script or update an existing one for clarification or in response to program changes. CMS officials approve scripts that are developed by the primary contractor, and check them for accuracy and completeness. The 1-800-MEDICARE help line provided accurate answers to 61 percent of the 420 calls we made. The accuracy rate varied significantly among the six questions we posed. Overall, 29 percent of our calls were answered inaccurately. In general, CSRs erred because they did not understand enough about the Medicare program to access the script with information to answer the question or clearly explain the material in it. In addition, for 10 percent of the calls we placed, we were unable to get a response to our question at the time we contacted the 1-800-MEDICARE help line, mainly due to problems when CSRs transferred calls. In response to our calls to the 1-800-MEDICARE help line, CSRs answered our questions accurately for 256 out of 420 calls, a rate of 61 percent. The criteria we developed to determine the information that constituted an accurate answer for each question are shown in table 1. (A more detailed version of the questions and information to answer them are in apps. II through VII.) The criteria were based on answers we developed from information on the Medicare Web site and were confirmed by CMS, which provided us with the scripts that contained information to answer the questions. We considered all calls we placed to the 1-800-MEDICARE help line to be part of our test of its accuracy, even if the call was transferred to a claims administration contractor to provide the answer. The percentage of calls CSRs answered accurately varied by question, as shown in figure 1. For example, CSRs accurately answered 81 percent of the calls asking whether a beneficiary could receive a prescription drug discount card if they had a Medigap policy. The answer to the Medigap question was clearly described in a script, which allowed CSRs to respond with the highest accuracy rate for all of our questions. Similarly, for question 1—choosing a prescription drug discount card—CSRs answered accurately 76 percent of the time. By July 2004, when we placed our calls, a large number of CSRs had been recently hired and trained specifically to answer this question, using a script and a Web-based tool. In contrast, for question 2 calls about the $600 prescription drug credit, CSRs answered inaccurately 79 percent of the time. CSRs scored poorly on this question primarily because they based their answers on the beneficiary’s total income without considering that some specific types of income should not be included in the calculation of eligibility for the credit. CSRs would have had to access two scripts to correctly answer the question, because the more general script on the topic did not contain all of the needed information. Question 5, which addressed Medicare part B enrollment, also had a relatively high inaccuracy rate—41 percent. We were not able to obtain an answer to some of our questions at the time that we called, most commonly when CSRs or the interactive voice response system transferred calls concerning questions 4 and 6 to other help lines. CSRs responded inaccurately to our questions largely because they did not seem to know enough to effectively use the scripts. According to a CMS official and the primary contractor’s staff, CSRs are expected to use scripts to guide their discussion with callers; they are not supposed to rely solely on acquired knowledge of Medicare to answer questions. We found, however, that in responding to our questions CSRs usually had one of four problems using scripts. The CSRs (1) did not seem to access a script, even when one was available; (2) did not seem to access a script with the right information to answer the question; (3) did not obtain enough information from the script; or (4) misunderstood some of the words in the scripts. We found instances when CSRs did not seem to access scripts when responding to calls. For example, when responding to our calls concerning the prescription drug discount card question, 2 CSRs indicated that they were not able to inform the caller about which card had the lowest drug prices—even though 53 other CSRs successfully used a script and a Web- based tool to answer this question. One other CSR referred our caller to AARP for an answer, rather than respond with the appropriate script and Web-based tool. These 3 CSRs did not seem to know how to correctly answer this question, which was addressed by one of the most commonly accessed script for the first half of the year. During 20 of the calls to answer our question on whether a spouse should enroll in Medicare part B if he had current employment-based health insurance, CSRs told our callers that enrolling in Medicare was a personal decision and they could not answer the question, which we classified as an inaccurate answer. They did not seem to recognize that they could access a script that contained information designed to answer that question. CSRs sometimes seemed to be accessing the wrong script to answer our question. For example, in answering the question on whether a beneficiary could receive a prescription drug discount card if she had a Medigap policy, one CSR incorrectly stated that the caller needed to complete a survey before receiving an answer. There is a script available that provides the answer to the Medigap question, but the script does not mention a survey. This CSR seemed to be using a different script about the prescription drug discount card, which has the right information to answer our question on the best prescription drug discount card to choose. In some cases, CSRs did not obtain enough information from the scripts they were using to accurately answer the question we asked. For example, to answer our question concerning whether a beneficiary could qualify for the $600 credit toward prescription drug purchases, the CSR should consider the source, as well as the amount of the beneficiary’s income. Some sources of income are not counted in determining a beneficiary’s eligibility for the $600 credit. According to CMS, to answer this question accurately the CSR would have to access two different scripts. The first script provides general information about the $600 credit and refers CSRs to the second script, which lists the sources of income that are not included in the eligibility calculation. However, the CSRs who answered this question incorrectly in 55 calls—or 79 percent of the time—focused on the total amount of income and did not seem to seem to consider the sources of the income or to access and use information from the second script. In 14 of the calls—or 20 percent of the time—CSRs were able to answer this question correctly, because they did consider the sources and amounts of income that we indicated the beneficiary had. Finally, CSRs sometimes misinterpreted or did not understand the words they were reading from the scripts or other written materials. For example, to answer our Medigap question, a CSR incorrectly told the caller that the beneficiary would automatically receive a prescription drug discount card if enrolled in a Medigap plan. The CSR may have been confusing Medigap policies with Medicare managed care plans, because both are discussed in the script that answered this question. In another example, for our question related to power wheelchair coverage, a CSR misread the requirement that a beneficiary must have adequate trunk—or upper body—strength. The CSR mistakenly informed us that a Medicare beneficiary needs to have adequate “trunk space” in order to qualify for a power wheelchair. When we asked for clarification, the CSR stated that Medicare requires a qualifying beneficiary to have adequate trunk space in his or her car to hold a power wheelchair. Similarly, during one of our calls about eye exam and glasses payment, the CSR informed us that an eye exam would be covered and then stated, “the only part of the exam that is not covered is ‘refraction,’ but I don’t know exactly what that is.” Because the CSR did not understand that a typical eye exam would be considered a refraction, she gave the caller the incorrect impression that Medicare would pay for a routine eye exam. CMS and contractor staff acknowledged that scripts for the 1-800- MEDICARE help line are not routinely pretested to ensure that both the CSR and the caller can understand the script before it is used to answer callers’ questions. On occasion, the 1-800-MEDICARE contractor has obtained CSRs’ feedback on draft scripts before they are used on the 1-800-MEDICARE help line to ensure that scripts can be easily read and understood. But this is not done as a routine step before new and revised scripts are used in handling calls. In addition, even if the CSRs consider the script understandable, it may still be confusing to Medicare beneficiaries. We found that pretesting to ensure that written material is understandable to its intended audience is a standard practice used to develop effective communications materials. For example, prior to issuing the first Medicare & You handbook nationwide to beneficiaries, CMS conducted consumer testing of its publication to evaluate its effectiveness as a communication tool. CMS has revised subsequent editions of the handbook to make it easier to read and use, based on feedback from beneficiaries. Moreover, other HHS agencies, such as the Centers for Disease Control and Prevention and the Substance Abuse and Mental Health Services Administration, have developed guidance on steps for ensuring that written material is understandable for intended readers and pretesting the materials before use. We did not receive answers to our questions for 10 percent (42) of the 420 calls we placed at the time we originally called. Several reasons accounted for this, as table 2 shows. For half (21) of the unanswered calls, the CSRs or the interactive voice response system transferred the calls placed during morning, evening, and weekend hours to claims administration contractors that were not open for business at the time of our call. Although the 1-800-MEDICARE help line is open 24 hours a day, 7 days a week, these other help lines are not. The transferred calls pertained to our questions concerning Medicare coverage about power wheelchairs and eye exams and glasses. The 1-800-MEDICARE CSRs or the interactive voice response system transfer such questions to the claims administration contractors’ help lines because these contractors generally have greater knowledge about Medicare coverage issues. Once our calls were transferred to closed help lines, we generally heard recordings that stated the contractors’ regular business hours and suggested calling back at that time. However, for 7 of those 21 calls, the contractors’ recorded messages did not provide a telephone number to use to call back during the stated business hours. The second most common reason we did not receive answers to our calls was that our calls were disconnected. Sixteen of the 42 unanswered calls were disconnected. For example, calls were disconnected before we were able to obtain a response to our questions. In one instance, the call was placed on hold for 30 minutes and then was disconnected. Four calls made on the same day did not receive a response because computer maintenance prevented the CSRs from accessing the Web-based tool they needed to use to answer our question about the prescription drug discount card. Finally, one other call was unanswered because the call was routed to a wrong telephone number. As required by CMS, both newly hired and experienced CSRs receive training to help them answer questions posed on the 1-800-MEDICARE help line. The training for newly hired CSRs includes instruction on accessing and using scripts, customer service etiquette, and information about the Medicare program. As part of the training, CMS requires newly hired CSRs to score 90 percent or higher on a written exam before they handle calls on the help line. All CSRs also receive continuing training, and take written quizzes on the new material. Although the 1-800-MEDICARE contractors met CMS’s training requirements by providing instruction and testing, the testing does not fully measure CSRs’ ability to accurately answer real questions from callers. The primary contractor develops and conducts the training new CSRs receive. Most of the training consists of 2 weeks of classroom instruction. In general, the instruction introduces CSRs to scripts and provides general information about the Medicare program. For example, in a training session we observed for newly hired CSRs in June 2004, the instructors helped the CSRs prepare for the types of inquiries that might be expected from callers on the 1-800-MEDICARE help line. The instructors posed different questions to the class, and each CSR attempted to identify and access a script with the right information to answer the instructor’s question. One CSR would be selected to read the script that they chose, and participants discussed whether they thought this was the script with the best information to answer the question. After completing their initial instruction, CMS requires the new CSRs to score at least 90 percent on a written exam and successfully complete a call handling simulation exercise before they answer calls on the help line without supervision. To successfully complete the call handling simulation exercise, CSRs must accurately answer two consecutive simulated help line calls out of six possible attempts. In addition, new CSRs generally spend about 4 hours listening to calls answered by an experienced CSR. In addition to the initial training for newly hired CSRs, CMS requires all CSRs to receive continuing training. Continuing training is delivered through three methods: refresher classes, online broadcast announcements, and small group meetings. Weekly refresher classes provide a means of instructing CSRs about Medicare program changes. Following each refresher training class, CSRs complete a short quiz to show that they understand the new information. While there is no minimum score that CSRs must achieve on the short quiz, CMS staff informed us that help line supervisors review each quiz to ensure that any questions that posed problems for CSRs would be addressed with further training. To provide CSRs with information quickly, the primary contractor sends online broadcast announcements to each CSR’s computer workstation. These online announcements usually contain information that may affect CSRs’ responses to help line questions, such as news about a change in a specific script. Lastly, small group meetings of about 12 CSRs and their supervisor are held for 30 minutes each week so that CSRs can discuss topics that can help them improve their call handling skills. After gaining experience in answering calls, some CSRs receive 4 additional days of special training and are promoted to a senior position. These CSRs receive classroom training on using Web-based computer programs that can assist Medicare beneficiaries in selecting a managed care plan, a nursing home, or other Medicare-related services. Like other CSRs, they must score 90 percent or higher on a written exam, and successfully complete a simulated call handling exercise before they can handle calls using the Web-based computer programs. Currently, about 200 senior CSRs answer calls on the 1-800-MEDICARE help line. Although all CSRs receive training and are tested as required, the responses we received indicate that not all CSRs had the necessary knowledge and skills to answer our questions accurately. In our opinion, testing how effectively CSRs use scripts to answer frequently asked questions provides the best measure of their preparation to do so. While 24 of the exam’s 52 questions ask CSRs to identify scripts that could be used to answer specific inquiries, the remaining 28 questions target other skills. In addition to the written test, new CSRs must appropriately answer questions posed in two consecutive simulated calls before they staff the help line. This simulated call handling and some of the written exam questions are the only measures of the CSRs’ ability to accurately answer calls using scripts. In combination, the test and the two simulated calls do not appear to be a sufficient measure of new CSRs’ ability to accurately answer the most frequently asked questions, given our findings on the accuracy of their responses. Further, while all CSRs receive continuing training, they are not required to demonstrate that they have effectively mastered the new material in handling calls. Developing a more targeted assessment of where CSRs need to augment their skills helped focus another help line’s training efforts and allowed it to meet its accuracy goals. In 2001, we assessed the telephone help line maintained by the IRS to answer taxpayers’ questions and found that it had not met the agency’s goals for accurately answering general questions about tax law and specific questions about individuals’ tax returns in 2001. In response, the IRS analyzed the specific types of inquiries within the area of tax law and individual returns that were answered inaccurately and identified the knowledge and skills its CSRs needed to answer questions more accurately. The IRS also identified the CSRs most in need of training to improve accuracy in those knowledge and skill areas and provided additional training to them before call volume increased for the 2002 tax season. By the end of the training period, these CSRs were required to be certified by their managers as capable of providing correct responses to taxpayer questions. The IRS also assigned responsibility for selected tax law topics to individual call center managers, making them accountable for ensuring that CSRs were trained and could accurately address inquiries on these topics. After these initiatives were complete, we found that the help line had improved its accuracy enough to meet its 2001 goals. In the span of 1 year, the accuracy rate on answering tax law questions increased from 79 to 85 percent and the accuracy rate for answering questions about individuals’ tax returns increased from 88 to 91 percent. CMS monitors the 1-800-MEDICARE help line mostly by requiring its primary contractor to evaluate four individual conversations that each CSR has with callers each month. Based on these conversations, the primary contractor evaluates the performance of individual CSRs in several categories, including accuracy, and reports the overall results to CMS. CMS also occasionally directly monitors a small number of individual CSRs’ calls. However, the contractor’s and CMS’s monitoring does not systematically track the accuracy rates for commonly asked questions. As a result, the monitoring does not assess how accurately CSRs as a group answer particular questions, which could help CMS target additional training efforts. Two smaller evaluation efforts did focus on specific questions answered inaccurately, and these targeted monitoring efforts provided information that CMS used to improve CSR training and the scripts used on the help line. At the time of our review, CMS had delegated most of the responsibility for monitoring the accuracy of the 1-800-MEDICARE help line to the primary contractor, while maintaining oversight by reviewing the primary contractor’s results. To monitor 1-800-MEDICARE, the primary contractor focuses on the performance of individual CSRs, evaluating four calls per month for each person. The primary contractor evaluates either live conversations—known as blind monitoring—or recorded conversations on the help line, while viewing displays of the CSRs’ computer activity during calls. Viewing the CSRs’ computer activity allows the primary contractor’s staff to observe the scripts or other materials that CSRs access to answer callers’ questions. After monitoring a call, the primary contractor’s supervisory staff uses a checklist to evaluate the CSR’s response to the caller. Help line supervisors share the results with each CSR to help improve performance. The primary contractor provides monthly reports to CMS on the results of its monitoring. The primary contractor has a subcontractor, which is responsible for conducting some independent call monitoring, as well as reviewing the results of some of the primary contractor’s call monitoring. In addition to the four calls per CSR per month that CMS requires the primary contractor to monitor, the subcontractor monitors up to one additional call per month per CSR. The subcontractor reports its monitoring results monthly to CMS and the primary contractor. In addition to meeting CMS requirements, the amount of call monitoring per CSR approximates industry standards. A survey of 735 North American call centers that represent help lines in various industries, including telecommunications, financial services, and health care, found a wide variance in the number of calls monitored per month. The most commonly reported monthly monitoring frequencies were 4 to 5 calls per CSR or 10 or more calls per CSR. The evaluation checklist used by CMS’s contractor for monitoring calls indicates that a CSR’s performance should include certain components— such as using an appropriate greeting, showing respect to the caller, actively listening to the caller, responding accurately to the question, providing a complete response, using appropriate tone and speed, offering to provide additional information if necessary, and ending the call politely. The primary contractor’s staff uses the checklist to evaluate both the customer service skills and knowledge skills demonstrated during a call and classify these into one of four categories—“unacceptable,” “needs improvement,” “achieves expectations,” and “exceeds expectations.” CMS requires the primary contractor to reach a quality rating of “achieves expectations” or higher for at least 90 percent of the total number of CSR calls evaluated each month. The primary contractor evaluates a call as either “accurate” or “inaccurate,” and because accuracy is weighted more heavily than other components, a CSR cannot provide inaccurate information on a call and still have the call scored as “achieves expectations.” However, in contrast to our methodology for this report, a CSR can provide incomplete information—information that is correct but does not fully answer the question—and still have the call scored as “achieves expectations.” In addition, the evaluation checklist does not indicate the specific criteria used to determine a call’s accuracy. Although CMS’s main role in monitoring the help line is to review the efforts of the primary contractor, the agency also conducts some monitoring of CSRs on its own. Like the primary contractor, CMS occasionally uses blind monitoring to evaluate the performance of individual CSRs—listening to real-time calls and watching the scripts and other materials the CSRs use. CMS does not conduct blind monitoring routinely and document the results, and therefore, CMS staff could not provide us any information on the extent of this monitoring. According to a CMS official, the agency conducts blind monitoring on a “limited and as- needed” basis. Although CMS’s primary 1-800-MEDICARE contractor monitors the accuracy of individual CSRs, CMS does not use the regular monthly monitoring to identify trends in inaccurate responses by question. Specifically, the primary contractor does not routinely classify or categorize CSRs’ answers by specific question to identify the questions that collectively were answered less accurately. While routine information about a question’s accuracy rate could be used to target improvement efforts, CMS has only taken this approach twice in recent years. Both of these efforts were small compared to the primary contractor’s monitoring. The larger effort to monitor accuracy by question lasted 29 months and involved 300 calls a month, whereas the primary contractor evaluated about 7,350 calls in July 2004. CMS contracted for a study to evaluate the 1-800-MEDICARE help line’s accuracy in answering specific questions, but did not receive results quickly enough to immediately address problems. Beginning in January 2002 and until May 2004, the CMS contractor hired to assess the “Medicare & You” program placed about 300 calls per month to the 1-800- MEDICARE help line. These callers used a set of hypothetical scenarios to assess how specific questions were answered. This study also established criteria specifying the information an accurate answer should provide and made a distinction between fully responsive answers—in other words, complete and accurate answers—or partially responsive answers—not complete but providing some accurate information. For the first 19 months studied, the average percentage of calls that received fully responsive answers ranged from under 40 percent to over 90 percent, depending on the question and the period of time studied. The study helped CMS identify questions that the CSRs were answering less accurately. However, CMS staff told us that the agency received the reports 4 to 5 months after monitoring occurred, which did not allow CMS to immediately address any identified problems. Nevertheless, CMS staff indicated to us that the results of these evaluations were used to identify areas where CSR refresher training was needed. Due to funding constraints, this project was terminated in May 2004. CMS told us it planned to resume a similar project in November 2004 around the time when the next cycle for the “Medicare & You” program contract begins. In another study, CMS measured 1-800-MEDICARE’s accuracy by question rather than individual CSR and found that certain questions were answered more accurately than others. In April 2004, a private consultant group that was under contract with CMS placed 49 calls to 1-800- MEDICARE to determine whether CSRs were relaying accurate information about Medicare’s new prescription drug discount card and benefit. The study established specific criteria on the information that CSRs should include for an answer to be accurate. Evaluating accuracy by question, the study found that CSRs accurately answered between 39 percent and 69 percent of the questions asked about the new Medicare prescription drug discount card and benefit. In contrast, in that same month the subcontractor—using its evaluation checklist—determined that the overall accuracy rate for the calls it measured on all topics to be about 94 percent. As a result of this study, CMS improved its scripts and related training. The private consultant’s report indicated that CSRs were having difficulty distinguishing between the Medicare prescription drug coverage benefit that will be in effect in 2006 with the Medicare prescription drug discount card that is currently available. CMS responded by clarifying the scripts used to answer these questions and improving the related materials used to train CSRs. For example, CMS worked with the contractor to rename the titles of the different scripts to include the term “benefit” or “card” as a method of differentiating between them. When the subcontractor noted about 3 months later that a few CSRs were continuing to confuse the prescription drug discount card and prescription drug benefit, CMS further clarified the scripts and its primary contractor conducted additional refresher training to attempt to correct the problem. Each year, millions of Medicare beneficiaries, their family members, and other callers rely on the 1-800-MEDICARE help line for information. Providing them with accurate answers is critical to keeping them informed about Medicare’s benefits. However, we found that 6 out of 10 calls were answered accurately, 3 out of 10 calls were answered inaccurately, and we were not able to get a response for 1 out of 10 calls. To answer inquiries accurately, CSRs have to be able to effectively access and use scripts. Given the lack of prior Medicare knowledge among CSRs, the 1-800-MEDICARE help line’s script-based approach is a reasonable means to facilitate accurate and consistent responses to caller’s questions. However, this approach makes CSRs—and thus the help line they support—dependent on the clarity and accuracy of the scripts available. Pretesting scripts might have identified ones that were difficult to understand by either the CSR or by potential callers, but this is not routinely done. Further, the training that CSRs receive on using scripts is also essential to their ability to answer questions accurately. However, the written exam that newly hired CSRs must pass and the continuing training quizzes do not measure the ability to use information in scripts to provide accurate and complete answers on the help line. Monitoring the help line could identify areas where CSRs’ knowledge and skills are lacking. Although CMS ensures that the amount of the contractor’s monitoring per CSR falls within industry standards, the bulk of the monitoring methods are not designed to systematically assess how accurately CSRs as a group answer particular questions. Evaluating how accurately particular questions are addressed is an important step to improving scripts and CSR training for those topics. Finally, 1-800-MEDICARE is advertised as providing information 24 hours a day, 7 days a week, but we could not always obtain answers to our questions when we called. When we called with questions about Medicare payment for power wheelchairs and coverage of eye exams and glasses, the help line frequently transferred our calls to claims administration contractors that were closed at the time. For a third of these transferred calls, we were not given a call-back number. This practice of transferring calls to claims administration contractors that are closed, in effect, reduces the benefit of a 24-hour help line to a business-hour help line for many beneficiaries. In order to improve the accuracy of responses made on the 1-800- MEDICARE help line and callers’ ability to have their questions answered, we recommend that the CMS Administrator take four actions: Assess the current scripts for the most commonly asked questions to ensure that they are understandable to CSRs and potential callers and if not, revise them as needed and pretest new and revised scripts to ensure that CSRs can effectively use them to accurately answer callers’ questions. Enhance testing of CSRs’ skills in accurately answering the most commonly asked questions using scripts and, if needed, provide additional training to improve the accuracy and completeness of their responses. Supplement current monitoring efforts to include a systematic review of the accuracy of information provided by the CSRs as a group for the most frequently asked questions and use the results to modify scripts or provide more training, as needed. Revise routing procedures and technology to ensure that calls are not transferred or referred to claims administration contractors during nonbusiness hours. In its written comments on a draft of this report, which are reprinted in appendix VIII, CMS agreed with our recommendations and stated that it had begun several efforts to address them. CMS also provided more detail on the challenges it faced in administering 1-800-MEDICARE due to the massive increase in call volume that occurred after the passage of the MMA. CMS agreed with our recommendation to assess current scripts and pretest new and revised scripts to ensure that they are understandable. In its comments, CMS stated that the written information used to develop 1-800-MEDICARE scripts often comes from Medicare publications that have been consumer tested as part of the publication preparation process. Language that has undergone some consumer testing is often incorporated into the scripts to improve clarity. While this step may be helpful, we believe that pretesting scripts verbally is also important, as consumer testing of material intended for written publication may not be adequate to determine whether the scripts are understandable to CSRs and the public. CMS also stated that it is considering implementing an editorial board to review scripts, which we believe would be another positive step to help assure the scripts’ clarity. CMS agreed with our recommendation to enhance testing of CSRs’ ability to accurately answer questions and provide additional training, as needed. CMS indicated that it was reassessing its testing requirements to determine better ways to ensure that CSRs are prepared to handle calls, once they are certified. The agency stated that it planned to benchmark its efforts against industry standards to determine more effective approaches. In its comments, CMS expressed concern that we had characterized customer service skills as less meaningful than knowledge skills. While we believe both are important, in keeping with our congressional mandate, this report focused on the accuracy of information provided by 1-800- MEDICARE and did not address the quality of its CSRs’ customer service skills. CMS agreed with our recommendation to supplement its current monitoring efforts by including a systematic review of the accuracy of information provided by CSRs as a group for the most frequently asked questions and using the results to modify scripts and provide more training, as needed. CMS indicated that it believed it had done a good job developing a quality assurance program that focuses on the most important requirements for both accuracy and customer service skills needed to answer calls from the elderly population. We agree that CMS has focused on quality assurance for 1-800-MEDICARE. Our recommendation did not deal with changing, but with enhancing, its quality assurance efforts. To address our recommendation, CMS indicated that it would implement a plan to develop trend information on the results of its quality assurance activities and would focus on improving the accuracy of responses to frequently asked questions. CMS agreed with our recommendation to revise procedures so that calls are not transferred to claims administration contractors during their nonbusiness hours. The agency indicated that although the 1-800- MEDICARE CSRs are available 24 hours a day, 7 days a week, they do not have access to claim-specific information. Therefore, the 1-800- MEDICARE CSRs would have to direct callers asking questions about specific claims to contact the claims administration contractors during their normal business hours. Our report focused on questions for which the CSRs had scripted responses and did not need to access claim-specific information. Nevertheless, for 7 of the 21 calls that were routed to claims administration contractors that were closed at the time we called, the contractors’ recorded messages did not provide a telephone number to call back during stated business hours. In its comments, CMS indicated that it had implemented additional routing plans to address this problem and is expanding access to claims data that will help reduce this problem in the future. CMS also raised concerns that we did not release the detailed audit documentation on our test calls while our work was still in progress. GAO’s policy does not allow us to provide audit documentation to an agency while work is ongoing. At the time of CMS’s request, we described our policy and offered to verbally provide more detail on telephone disconnections, but CMS did not follow up with us to obtain this information. After our report is published, we will address this request. Finally, CMS expressed concern that we did not describe the criteria we used to evaluate the accuracy of responses to our six questions and stated that incomplete answers should not be considered inaccurate responses. As noted in the draft report, table 1 lists the criteria that we established for each of our six questions. We developed these criteria so that we could objectively evaluate responses received from CSRs. For four of these questions (numbers 1, 2, 3, and 6), there was only one element in the correct response, so an incomplete response was not possible. For the remaining two questions, we considered the answer accurate if it included two elements. We believe that by not including both elements for each of these questions, callers would be left with a false impression, rather than with an accurate answer. For example, in evaluating the response to the question of whether Medicare would pay for a power wheelchair, we thought it was important for the caller to know that (1) the wheelchair needed to be prescribed by a physician and (2) the beneficiary would be responsible for a copayment. Because the copayment for a power wheelchair is at least $1,000, we believe that it would be misleading not to mention either a copayment or cost sharing when a caller asks whether Medicare pays for this item. Likewise, needing to have a physician prescribe the power wheelchair is a Medicare requirement, and we did not think a response could be accurate without mentioning it. For the question on Medicare part B enrollment, we thought that it was important to know that a beneficiary could wait to enroll but, once other health insurance coverage ended, had a limited time period to enroll in part B without incurring higher premiums. Without knowing both elements of this answer, beneficiaries would not have enough information to guide their decision on part B enrollment and, therefore, the answer provided would be misleading. We are sending copies of this report to the Administrator of CMS, appropriate congressional committees, and other interested parties. We will also make copies available to others upon request. This report is also available at no charge on GAO’s Web site at http://www.gao.gov. If you or your staff have any questions about this report, please call me on (312) 220-7600. An additional GAO contact and other staff who made contributions to this report are listed in appendix IX. To determine the accuracy of information provided, we placed a total of 420 calls to the 1-800-MEDICARE help line. We made 420 calls in order to have a sample that was large enough to determine if differences in accuracy were significant. We selected six questions about Medicare— three related to the Medicare prescription drug discount card and three related to Medicare coverage or eligibility for benefits. We asked each of the selected questions a total of 70 times. We randomly placed calls at different times of the day and different days of the week between July 8 and July 30, 2004, to match the daily and hourly pattern of calls reported by 1-800-MEDICARE in April 2004. To select the 6 questions, we initially chose 20 questions that related to the 100 topics most frequently addressed by the 1-800-MEDICARE help line’s CSRs in May 2004 and developed criteria for an accurate response from information on the Medicare Web site’s frequently asked questions section. We then presented the 20 questions and answers to Centers for Medicare & Medicaid Services (CMS) officials, who provided us with a script number and text for each question. CMS officials did not object to using any of the 6 questions that we ultimately chose, or suggest that the answers that we had provided for these questions were incorrect. We informed both CMS and one of the 1-800-MEDICARE contractors that we would be placing these calls. However, we did not tell them which 6 of the 20 questions we selected, or the specific dates and times when we would be placing our calls. Before placing our calls, we created a scenario with fictional names and zip codes for each of the six questions to make them sound more realistic. (Appendices II to VII contain the scenarios that we used.) We made pretest calls for each question before we finalized its wording. During our actual calls, the CSRs were not aware that their responses would be included in a research study. We recorded the length of, and routing process for, each call. We evaluated the accuracy of the responses by CSRs to the 420 calls we placed by whether key information was provided. The results from our 420 calls are limited only to those calls and are not generalizable to the population of calls routinely made to call centers by beneficiaries or other callers. Although the six questions we posed were among topics most often accessed by CSRs, they do not encompass all of the questions callers might ask. In addition, we did not verify the reliability of CMS’s monitoring data. To examine the training provided to 1-800-MEDICARE CSRs, we interviewed officials representing CMS and the 1-800-MEDICARE contractor responsible for training CSRs. We reviewed the primary contractor’s training requirements and the instructional materials that are used to educate new CSRs. We also observed a training session for new CSRs at a 1-800-MEDICARE call center. In addition, we reviewed previous GAO reports on the operations of other help lines, including the training provided to the CSRs answering calls on the Internal Revenue Service’s help line. To evaluate CMS’s role in overseeing the accuracy of information provided through the 1-800-MEDICARE help line, we interviewed officials from CMS and one of the two 1-800-MEDICARE contractors about their monitoring and oversight activities. For our first objective, we focused on the accuracy of information provided by the CSR regardless of the contractor who managed their work. For our other two objectives, we relied on information provided by one of the two contractors—which we refer to as the primary contractor. We also identified CMS requirements for call center operations and reviewed contractor reports to identify the types of problems encountered through the help line. We performed our work from May 2004 through December 2004 in accordance with generally accepted government auditing standards. For question 1, which is about choosing a prescription drug discount card, we used scenarios with different combinations of prescription drugs and one of four different locations, in order to ensure our anonymity. If a CSR named one of the two prescription drug discount cards with the lowest cost for the combination of prescription drugs in the scenario posed, we considered it to be a correct response to our question. To ensure that we obtained the correct answer for each question, we periodically checked the prescription drug prices using the prescription drug tool on the Medicare.gov site. This is the same tool CSRs used to answer our questions. The answers shown in this appendix were accurate as of July 15, 2004. Question 1a posed to CSRs: My father-in-law lives in Wayne, Pennsylvania, and wants to continue to shop at Yorke Apothecary (located at 110 S. Wayne Ave., Wayne, Pennsylvania, 19087). What drug card can he get that will cover all of his drugs at Yorke Apothecary, and cost the least amount? He takes the following drugs: Other information to provide to the CSR if asked: He is single. He lives in Wayne, Pennsylvania, in Delaware County. His zip code is 19087. He currently has fee-for-service Medicare with no other drug benefits. He does not use an American Indian Health pharmacy. He does not live in a long-term care facility. He has $20,000 in annual income and is not interested in any drug assistance programs, including the $600 credit. His sources of income are a pension and Social Security, but the amount from each is unknown. He has some bank accounts, but their value is unknown. The amount he currently pays for drugs is unknown. Default answer for other questions: “I don’t know.” Two prescription drug discount cards listed on Medicare.gov with the lowest prices for the combination of drugs in our scenario: myPharmaCare Prescription Drug Discount Card 1-800-601-3002 Monthly drug costs: $116.69 Annual enrollment fee: $25.00 U Share Prescription Drug Discount Card 1-800-707-3914 Monthly drug costs: $115.59 Annual enrollment fee: $19.95 Question 1b posed to CSRs: My father lives in Homewood, Illinois, and wants to continue to shop at the K-Mart Pharmacy in Homewood, Illinois (located at 17550 Halsted Rd., Homewood, Illinois, 60430). What drug card can he get that will cover all of his drugs at the K-Mart Pharmacy, and cost the least amount? He takes the following drugs: Other information to provide to the CSR if asked: He is single. He lives in Homewood, Illinois, in Cook County. His zip code is 60430. He currently has fee-for-service Medicare with no other drug benefits. He does not use an American Indian Health pharmacy. He does not live in a long-term care facility. He has $20,000 in annual income and is not interested in any drug assistance programs, including the $600 credit. His sources of income are a pension and Social Security, but the amount from each is unknown. He has some bank accounts, but their value is unknown. The amount he currently pays for drugs is unknown. Default answer for other questions: “I don’t know.” Two prescription drug discount cards listed on Medicare.gov with the lowest prices for the combination of drugs in our scenario: U Share Prescription Drug Discount Card 1-800-707-3914 Monthly drug costs: $174.91 Annual enrollment fee: $19.95 Any of several prescription drug discount cards available with this combination of drugs priced at $182.80. Question 1c posed to CSRs: My father lives in Cincinnati, Ohio, and wants to continue to shop at the CVS Pharmacy (located at 3195 Linwood Ave., Cincinnati, Ohio). What drug card can he get that will cover all of his drugs at the CVS Pharmacy, and cost the least amount? He takes the following drugs: Other information to provide to the CSR if asked: He is single. He lives in Cincinnati, Ohio, in Hamilton County. His zip code is 45226. He currently has fee-for-service Medicare with no other drug benefits. He does not use an American Indian Health pharmacy. He does not live in a long-term care facility. He has $20,000 in annual income and is not interested in any drug assistance programs, including the $600 credit. His sources of income are a pension and Social Security, but the amount from each is unknown. He has some bank accounts, but their value is unknown. The amount he currently pays for drugs is unknown. Default answer for other questions: “I don’t know.” Two prescription drug discount cards listed on Medicare.gov with the lowest prices for the combination of drugs in our scenario: myPharmaCare Prescription Drug Discount Card 1-800-601-3002 Monthly drug costs: $202.79 Annual enrollment fee: $25.00 Anthem Prescription Drug Discount Card 1-800-730-2804 Monthly drug costs: $209.87 Annual enrollment fee: $14.95 Question 1d posed to CSRs: My father lives in Brooklyn, New York, and wants to continue to shop at the Neergaard Pharmacy (located at 454 Fifth Avenue, in Brooklyn, New York). What drug card can he get that will cover all of his drugs at the Neergaard Pharmacy, and cost the least amount? He takes the following drugs: Other information to provide to the CSR if asked: He is single. He lives in Brooklyn, New York, in Kings County. His zip code is 11215. He currently has fee-for-service Medicare with no other drug benefits. He does not use an American Indian Health pharmacy. He does not live in a long-term care facility. He has $20,000 in annual income and is not interested in any drug assistance programs, including the $600 credit. His sources of income are a pension and Social Security, but the amount from each is unknown. He has some bank accounts, but their value is unknown. The amount he currently pays for drugs is unknown. Default answer for other questions: “I don’t know.” Two prescription drug discount cards listed on Medicare.gov with the lowest prices for the combination of drugs in our scenario: EnvisionRx Plus Prescription Drug Discount Card 1-866-250-2005 Monthly drug costs: $46.33 Annual enrollment fee: $30.00 Any of several prescription drug discount cards available with this combination of drugs priced at $50.45. Question 2 posed to CSRs: I’ve heard about the $600 credit that can help pay for prescriptions and wanted to know if my mother was eligible for it. Could she qualify for the credit? I know she has three sources of income. She has about $765 per month from Social Security. She also gets $250 each month in rental income from the apartment in the downstairs part of her house. She has a tenant that pays rent to her. She’s also getting a payout from my father’s life insurance policy of $70 each month. Other information to provide to the CSR if asked: She is single and lives alone. She only has fee-for-service Medicare as health insurance. She owns her house. She lives in Miami, Florida, 33129. Default answer for other questions: “I don’t know.” Information from Medicare.gov that GAO used to develop accuracy criteria: If your annual gross income is currently no more than $12,569 ($1,048 per month) as a single person or no more than $16,862 ($1,406 per month) for a married couple, you might qualify for a $600 credit to help pay for your prescription drugs and Medicare may pay your annual enrollment fee. If you and your spouse both qualify for the credit, the credit will be put on each of your cards. TRICARE for Life provides secondary military health coverage available for Medicare- eligible uniformed services beneficiaries, their eligible family members, and survivors enrolled in Medicare part B. The following sources of income should be included when calculating your gross income for your $600 credit enrollment form: Employee compensation (salary, wages, tips, bonuses, awards, etc.) Unemployment compensation Pensions and annuities Social Security benefits (including Social Security Equivalent portion of Railroad Retirement) Railroad Retirement benefits Veterans Affairs benefits Military and government disability pensions – armed forces, Public Health Service, National Oceanic and Atmospheric Administration, Foreign Service (based on date pension began, combat-related pension, etc.) Individual Retirement Account distributions Interest (savings accounts, checking accounts, etc.) Ordinary dividends (stocks, bonds, etc.) Refunds, credits, or offsets of state and local income taxes Alimony received Business income Capital gains Farm income Rental real estate, royalties, partnerships, trusts, etc. Other gains (sale or exchange of business property) Other income (lottery winnings, awards, prizes, raffles, etc.) The following sources of income should not be included when calculating your income for $600 credit enrollment form: Inheritances and gifts (taxed to estate or giver if not under limits for exemption) Interest on state and local government obligations (e.g., bonds) Workers compensation payments Federal Employees Compensation Act payments Supplemental Security Income benefits Income from national senior service corps programs Public welfare and other public assistance benefits Proceeds from sale of a home Lump sum life insurance benefits paid upon death of insured Life insurance benefits paid in installments Accelerated life insurance death benefit payments (e.g., viatical settlements, terminal illness, chronic illness) Question 3 posed to CSRs: I’m calling with a question about my grandmother. She is 69 and she has Medicare, and she also has a Medigap policy. Could you please tell me if she can still get a Medicare-approved drug discount card? Other information to provide to the CSR if asked: She is single and lives alone. She lives in Miami, Florida. I don’t know the zip code off-hand. She is not in a long-term care facility. She is enrolled only in Medicare fee-for-service. She doesn’t have a Medicare managed care plan. She is not enrolled in Medicaid. I don’t think she’s interested in the $600 credit right now; I was just wondering if she could get the prescription drug discount card. Default answer for other questions: “I don’t know.” Information from Medicare.gov that GAO used to develop accuracy criteria: Having a Medigap policy does not preclude a Medicare beneficiary from being eligible for a Medicare prescription drug discount card. For question 4 about power wheelchairs, we provided the CSRs with one of four different city and state combinations, as shown below. The four city and state combinations were randomly assigned to different power wheelchair calls. We did this to ensure that if our call was transferred to one of the four claims administrator contractors that administer Medicare’s durable medical equipment claims—including power wheelchair claims—we were not biasing our results toward any particular claims administrator. Question 4 posed to CSRs: My father is having trouble getting around. He has a hard time walking and doesn’t have much upper body strength. Could you please tell me if Medicare will pay for a power wheelchair for him? Other information to provide to the CSR if asked: He is enrolled in Medicare, both parts A and B. He lives in : Philadelphia, Pennsylvania. His zip code is 19105. Detroit, Michigan. His zip code is 48209. Pensacola, Florida. His zip code is 32516. Scottsdale, Arizona. His zip code is 85262. His doctor has suggested he get a power wheelchair to improve his mobility. He doesn’t have enough strength to use a manual wheelchair. He lives alone and is not married. Default answer for other questions: “I don’t know.” Information from Medicare.gov that GAO used to develop accuracy criteria: Power wheelchairs and/or scooters are covered if they are medically necessary based on Medicare’s criteria for coverage. In order for Medicare to cover a power wheelchair/scooter, the beneficiary’s doctor must provide a prescription or certificate of medical necessity that states that he needs it because of his medical condition. If your father qualifies for coverage, Medicare will pay 80 percent of the Medicare-allowed amount. Question 5 posed to CSRs: Should my husband sign up for part B if I am still working and we have health insurance coverage from my employer? Other information to provide to the CSR if asked: My husband is about to turn 65 next January. If asked whether working for a large or small employer: I work for the federal government. I have full medical coverage, including dental and vision. My husband is fully covered under my insurance plan. Neither of us is disabled. The city/zip code information that corresponds with the location of the caller. Default answer for other questions: “I don’t know.” Information from Medicare.gov that GAO used to develop accuracy criteria: Your husband might want to wait to sign up for part B, because he would have to pay the monthly part B premium and the benefits may be of limited value as long as the group health plan is the primary payer. You could save on monthly premiums by waiting to sign up. If your husband doesn’t sign up for part B when first eligible because he has group health coverage through an employer, he can sign up for Medicare part B during a special enrollment period. This can be anytime he is still covered by the employer’s group health plan or during the 8 months following the month when either the coverage or the employment ends—whichever is first. Most people who sign up for Medicare part B benefits during a special enrollment period do not pay higher premiums. Question 6 posed to CSRs: My mother is 66 and is enrolled in Medicare. She has been complaining lately that she is having trouble reading the paper and thinks she may need new eyeglasses. Will Medicare pay for an eye exam and a new pair of eyeglasses if her prescription has changed? Other information to provide to the CSR if asked: The city/zip code information that corresponds with the location of the caller. She is not married. She is enrolled in Medicare fee-for-service only. I do not know the name of the county she lives in. Default answer for other questions: “I don’t know.” Information from Medicare.gov that GAO used to develop accuracy criteria: Medicare does not pay for routine eye exams, eyeglasses, or contact lenses. The beneficiary must pay 100 percent of these services. Shaunessye D. Curry, Joy L. Kraybill, Krister P. Friday, Sari B. Shuman, Mary W. Reich, Ramsey L. Asaly, Alexis Chaudron, Perry G. Parsons, and Leslie Spangler made key contributions to this report.

Summary: In March 1999, the Centers for Medicare & Medicaid Services (CMS) implemented a telephone help line--1-800-MEDICARE--to provide information about program eligibility, enrollment, and benefits. The Medicare Prescription Drug, Improvement, and Modernization Act of 2003 (MMA) directed GAO to examine several issues related to this 24-hour help line and the customer service representatives (CSRs) who staff it. In this report, GAO evaluated (1) the accuracy of the information the help line provides, (2) the training given to CSRs, and (3) CMS's efforts to monitor the accuracy of information provided through the help line. The 1-800-MEDICARE help line provided accurate answers to 61 percent of the 420 calls we made and inaccurate answers to 29 percent. We were not able to obtain any answers for the remaining 10 percent of our calls at the time we placed them. Most of these calls were not answered because they were transferred to other contractors responsible for processing Medicare claims that were not open for business at the time we called or these calls were inadvertently disconnected. To facilitate accurate responses, the 1-800-MEDICARE help line provides CSRs with written answers--called "scripts"--that CSRs use during a call. When CSRs provided inaccurate information, it was largely because they did not seem to access and effectively use a script that answered our questions. CMS and its contractor do not routinely pretest the scripts to ensure that they are understandable to CSRs or potential callers. The training for CSRs meets CMS's requirements, but it is not sufficient to ensure that CSRs are able to answer questions accurately on the help line. Before handling calls, CSRs must complete about 2 weeks of classroom training; accurately answer two simulated calls consecutively out of six; and score at least 90 percent on a written exam. In addition, all CSRs receive ongoing training. However, the results from our calls indicate that the testing and simulated call answering did not sufficiently measure whether CSRs were prepared to answer questions accurately. CMS delegates most accuracy monitoring to one of its contractors and reviews the results. The bulk of the monitoring focuses on how accurately individual CSRs answer questions. However, this monitoring does not systematically track questions answered inaccurately by CSRs as a group, which could help target training and script improvement. Through two smaller studies that measured how accurately specific questions were answered, CMS was able to identify areas to improve scripts and training.