PPACA directed each state to establish and operate a health insurance marketplace by January 1, 2014. In states electing not to establish a marketplace, the law required HHS (which delegated this role to CMS) to do so. These marketplaces were intended to provide a seamless, single point of access for individuals to enroll in private health plans and apply for income-based financial assistance established under the law. CMS reported that around 12.7 million individuals applied to enroll in healthcare coverage for plan year 2016. PPACA and HHS regulations and guidance required each marketplace to be able to carry out four key functions, among others: 1. Eligibility and enrollment. Assess and determine an individual’s enrollment eligibility, enroll eligible individuals in coverage, and certify private health insurance plans for participation in the marketplace. 2. Plan management. Provide services for activities such as submitting, monitoring, and renewing qualified health plans. 3. Financial management. Facilitate payments to health insurance issuers as well as provide services such as payment calculation for risk-adjustment analysis and cost-sharing reductions for individual enrollments. 4. Consumer assistance. Provide assistance to consumers in completing an application, obtaining eligibility determinations, comparing coverage options, and enrolling in coverage. To provide these capabilities, PPACA further required the states establishing marketplaces, as well as CMS, to design, develop, implement, and operate health insurance marketplace IT systems. States have some flexibility as to the marketplace types, such as: State-based marketplaces. These states developed their own marketplaces with final decision-making authority, and were provided full autonomy in setting user fees and establishing sustainability plans. The marketplaces varied by state, depending on each state’s current and previous health care systems environment, governance and business models, applicable laws and regulations, and other factors. State-based marketplaces using the federal platform. These states initially elected to develop their own state marketplace systems, but due to IT, financial, or other challenges, subsequently decided to use the federal platform to perform certain eligibility and enrollment functionalities. Issuers of health insurance in these states offer plans through the federal platform and are charged a user fee by CMS for the services and benefits they provide. These states also have the authority to collect user fees beyond those collected by CMS. Federal responsibilities, carried out by CMS, include managing the federal IT platform, call center infrastructure, and eligibility determinations. The states must maintain websites that provide information to consumers and direct them to HealthCare.gov to apply for and enroll in coverage. In addition, these states must coordinate with CMS on outreach strategies to reach existing and new consumers; maintain data and timely reporting for all coverage years prior to the transition; and work with issuers to ensure they are prepared to transition to the federal platform and exchange enrollment data with CMS. Federally facilitated marketplaces. These states elected not to develop their own platform and use the federally facilitated marketplace—the federal IT platform, including the website, HealthCare.gov. Federally facilitated partnerships. These states are a variation of the federally facilitated marketplaces, whereby CMS establishes and generally operates the marketplaces and the states assist CMS with operating various functions, such as plan management and consumer assistance. In these cases, states rely to varying degrees on the systems developed by CMS to support the federally facilitated marketplaces and the federal government keeps the user fees paid by the insurers. For plan year 2016, utilizing state and CMS documentation, we categorized 27 states as having federally facilitated marketplaces, 7 as having federally facilitated partnerships, 13 as having state-based marketplaces, and 4 as having state-based marketplaces using the federal platform. Figure 1 shows the type of health insurance marketplace used by each state for plan year 2016. To establish, operate, and sustain health insurance marketplaces, states have used several IT funding sources, which vary based on the marketplaces’ operational model and the IT work being performed. Marketplace grants: PPACA authorized CMS to award federal exchange (marketplace) grants through December 2014 for planning and implementation activities, as well as for the first year of a marketplace’s operation. The federal marketplace grant stages include: Pre-award. A funding announcement notified states about the grant opportunity. States then submitted applications for CMS review. Award. CMS identified successful applicants and awarded funding to be used during specific time periods. Implementation. Grantees draw down funds from preauthorized grant accounts monitored by CMS, and report financial and performance information to the agency. States are allowed to request a no-cost grant extension in order to use remaining approved federal funds to complete project goals and objectives. CMS reviews each no-cost extension request to ensure that expenses are allowable, correctly allocated, and reasonable based on PPACA and CMS grant rules and policies. States also can resubmit budget requests to CMS to authorize reallocation of grant funding among previously awarded budget categories. Closeout. This occurs after the period of performance ends and includes preparation of final reports, financial reconciliation, and any required accounting for property and funds. After a grant is closed, CMS deobligates any remaining amounts. (In this report, we use “deobligated funds” to describe amounts that are no longer available to CMS for new obligations, although they may remain available for certain limited other purposes.) Medicaid funding: PPACA enactment required changes to Medicaid eligibility and enrollment systems so that the program could operate seamlessly with the marketplaces, as well as to implement new Medicaid eligibility policies. Under federal law, states are eligible for an enhanced federal matching rate of 90 percent (referred to as 90/10 funding—states contribute 10 percent of the cost) for the design, development, or installation of Medicaid claims processing and information retrieval systems. Because states’ Medicaid systems had to be replaced or modernized to meet PPACA requirements, CMS expanded the availability of the 90/10 funding for states to make changes to improve Medicaid eligibility IT systems, including the connection to the federal marketplace. In addition, a state could receive funding in the form of a 75 percent federal matching rate for the maintenance and any ongoing costs of operating its upgraded Medicaid eligibility and enrollment system. The funding is generally available when the upgraded system becomes operational. States were also allowed to use Medicaid funds when they transitioned to a different marketplace type since the transition costs were related to updating state Medicaid eligibility systems to be compatible with the federal platform. Revenue through user fees: PPACA authorized state-based marketplaces to charge assessments, or user fees, such as a percentage or a flat monthly rate to participating insurance issuers. For the selected states to sustain their marketplaces, Hawaii charged a user fee of 3.5 percent per insurance plan, Minnesota charged a 3.5 percent user fee, and Oregon charged a user fee of $9.66 per member per month. New York does not impose a user fee. Additional revenue sources: In addition to user fees, states have leveraged their own funds to support and sustain marketplaces, including general funds, fund transfers from other agencies, and broad-based assessments. Examples of these additional revenue sources for the four selected state marketplaces include: Hawaii’s marketplace received funding from state general funds. Minnesota’s marketplace incurred costs associated with enrollment of individuals into public programs by the Minnesota Department of Human Services. Payments from the Minnesota Department of Human Services represent its share (in both federal and state funds) of enrollment costs in public programs. New York’s marketplace received revenue from a broad-based assessment funded by the Health Care Reform Act instead of from user fees. According to New York officials, that revenue included assessments of certain medical services and is used to support several health care programs in the state, including the New York marketplace. Oregon’s marketplace received reimbursements from other state agencies for shared software license fees. Operating reserves: Some states also implemented dedicated state financial reserves for marketplace operations, including IT. Reserves can cover increases in expenditures, such as unexpected development costs, or decreases in revenue, such as decreased user fees due to lower than expected enrollment. Minnesota and Oregon marketplaces were legislatively allowed to use such reserves. Minnesota had less than 1 month of reserves and Oregon’s marketplace had 12 months of reserves in 2016. Hawaii and New York did not allow their marketplaces to have reserves. Table 1 provides a summary of the four selected states’ marketplace revenue sources and reserves. To address the requirements of PPACA and its implementing policies, HHS designated CMS to provide oversight of the IT supporting states’ marketplaces, and CMS assigned three key offices to do so—the Center for Consumer Information and Insurance Oversight (CCIIO), the Center for Medicaid and CHIP Services (CMCS), and the Office of Technology Solutions (OTS). CMS relies on these offices to, among other things, perform the following reviews: Establishment reviews. CMS is to conduct establishment reviews of states that receive federal marketplace grant and Medicaid funds, following the Enterprise Life Cycle framework. These reviews are intended to show the progress that the states made in using federal funding to implement marketplace IT systems. The framework requires states to provide specific artifacts for CMS review, such as the concept of operations, system test documents, and project plans, among others. Each review is incremental and states are expected to show CMS an acceptable level of progress and maturity in their projects’ development before proceeding to the next project phase. Review of annual state financial/programmatic reports. State-based marketplace annual reporting tool (SMART): CMS collects and reviews state-based marketplace financial and programmatic data regularly through SMART. The agency collects and reviews annual financial/programmatic audit reports, and attestations of compliance as part of this process. CMS staff identify observations, action items, and ongoing monitoring activities that could improve marketplace operations. For example, they may identify areas where IT functionality needs improvement, or more clarity on IT expenditures in marketplace financial audit reports. Oversight and program integrity standards for state exchanges: Beyond SMART, CMS requires state-based marketplaces to also annually provide financial statements, eligibility and enrollment reports, and performance monitoring data. Frequent, regular communication. CMS is to monitor and provide assistance to states through frequent and regular communication, including weekly telephone calls with state officials involved with marketplace IT efforts. CMS includes subject matter experts in these calls, as needed. State officials can report concerns or provide further information during the weekly calls. The selected state marketplaces—Hawaii, Minnesota, New York, and Oregon—each had different experiences with regard to establishing and operating their marketplaces. Each state’s establishment, operations, and funding experiences are summarized in the discussions and figure 2 below, which are based, in part, on CMS budget summary documentation for the states as of October 2016. Since 2011, CMS had awarded Hawaii $205.3 million in federal marketplace grants to establish a state-based marketplace. As of October 2016, the state had spent or planned to spend $140.4 million of those grants, including $97.4 million for IT costs associated with building a health insurance marketplace and developing infrastructure needed for the marketplace’s ongoing operations. The Hawaii Health Connector implemented the original state-based marketplace, which began operation in 2013. In 2015, the marketplace ceased operations and transitioned to a state-based marketplace using the federal platform. According to CMS, the transition was due to a variety of system issues and other factors that limited enrollment and had a negative impact on the consumers’ use of the system. Hawaii decided to undergo an additional transition in June 2016 in order to operate as a federally facilitated marketplace for plan year 2017. As a result of the state’s marketplace transition efforts, CMS deobligated $63 million of Hawaii’s grant funds as of October 2016 for grants that had concluded their period of performance and had been closed. As of March 2016, approximately 15,000 individuals in Hawaii were enrolled in qualified health plans. Since 2011, CMS had awarded $189.4 million in federal marketplace grants to Minnesota to build a health insurance marketplace and develop infrastructure needed for the marketplace’s ongoing operations. MNsure had implemented the state-based marketplace and spent or planned to spend $159.6 million in grant funds as of October 2016, which included IT expenditures of $48.2 million. Additionally, CMS had deobligated $102,000 of Minnesota’s grant funds as of October 2016 for grants that had concluded their period of performance and had been closed. Minnesota is 1 of 12 states that have an integrated system for both Medicaid and qualified health plans and is 1 of 2 states that have a Basic Health Program. This allows Minnesota to combine eligibility determinations for qualified health plans, Medicaid, and the state Basic Health Program. As of March 2016, Minnesota had enrolled approximately 84,000 individuals in qualified health plans. Since 2011, CMS had awarded $575.1 million in federal marketplace grants to the state of New York to build a health insurance marketplace and develop infrastructure needed for the marketplace’s ongoing operations. The New York Health Benefit Exchange (later renamed the New York State of Health) within the New York State Department of Health implemented the state-based marketplace and had spent or planned to spend $513.6 million in marketplace grants as of October 2016, to include $209.2 million for IT costs. Additionally, CMS deobligated $4.5 million of New York’s grant funds as of October 2016 for grants that had concluded their period of performance and had been closed. New York is the other state, along with Minnesota, to have implemented a Basic Health Program. As of March 2016, New York State of Health had enrolled approximately 272,000 individuals in qualified health plans. Since 2011, CMS had awarded Oregon $305.2 million in federal marketplace grants to build a health insurance marketplace and develop infrastructure needed for the marketplace’s ongoing operations. Oregon had spent $301.4 million in federal marketplace grants as of October 2016, which included IT spending of $79.7 million. Additionally, CMS had deobligated $1.8 million of Oregon’s grant funds as of October 2016 for grants that had concluded their period of performance and had been closed. Cover Oregon implemented the original marketplace, which operated from 2013 to 2014 as a state-based marketplace. The state-based marketplace then transitioned to a state-based marketplace using the federal platform under the Oregon Department of Consumer and Business Services, due to the technical challenges of IT development and creating a functional website to enroll consumers. As of March 2016, approximately 147,000 individuals had selected a qualified health plan. The four selected states’ overall marketplace federal grant funding, expenditures, including amounts planned to be spent, and deobligations, as well as their enrollment as of plan year 2016 are reflected in figure 2. Over the past 4 years, we have issued various reports highlighting challenges that CMS and the states have faced in implementing and operating health insurance marketplaces. For example, in an April 2013 report, we described the actions of seven states that were in various stages of developing an IT infrastructure to establish marketplaces, including redesigning, upgrading, or replacing their outdated Medicaid and CHIP eligibility and enrollment systems. Six of the seven states were also building the IT infrastructure needed to integrate systems and allow consumers to navigate among health programs. However, the states had identified challenges with the complexity and magnitude of the IT projects, time constraints, and guidance for developing their systems. In December 2014, we reported that all states using the federal marketplace IT solution had faced challenges transferring applications to and from that system. We pointed out that none of the states using the federal marketplace IT solution in the first enrollment period were able to implement application transfers, which required the establishment of two IT connections: one connection to transfer applications found ineligible for Medicaid coverage from the state Medicaid agency to the federal marketplace IT solution, and another connection to transfer applications found ineligible for coverage from the federally facilitated marketplace to the state Medicaid agency. In March 2015, we reported that several problems with the initial development and deployment of HealthCare.gov and its supporting systems had led to consumers encountering widespread performance issues when trying to create accounts and enroll in health plans. We noted, for example, that CMS had not adequately conducted capacity planning, adequately corrected software coding errors, or implemented all planned functionality. In addition, the agency did not consistently apply recognized best practices for system development, which contributed to the problems with the initial launch of HealthCare.gov and its supporting systems. In this regard, weaknesses existed in the application of requirements, testing, and oversight practices. Further, we noted that CMS had not provided adequate oversight of the HealthCare.gov initiative through its Office of the Chief Information Officer. We made seven recommendations aimed at improving requirements management, system testing processes, and oversight of development activities for systems supporting HealthCare.gov. CMS concurred with all of our recommendations and subsequently took or planned steps to address the weaknesses, including instituting a process to ensure that functional and technical requirements are approved; developing and implementing a unified standard set of approved system testing documents and policies; and providing oversight for HealthCare.gov and its supporting systems through the department-wide investment review board. Further, in September 2015, we reported that states had spent $3.22 billion in federal grant funding to establish their health insurance marketplaces, of which approximately $1.45 billion was to establish IT systems supporting their health insurance marketplaces. However, we noted limitations in CMS’s oversight of the states’ IT system development, such as a lack of clearly defined roles and responsibilities leading to challenges with states receiving timely CMS guidance and relevant CMS senior executives not always being involved in state IT funding decisions. We also noted that CMS had allowed state systems to be launched without being fully tested. This led to consumers in some states experiencing long waits for eligibility determinations, websites freezing, system failures, and manual enrollments. We made three recommendations aimed at improving CMS oversight in those areas and CMS concurred with all of the recommendations, noting actions they had taken or planned to take in each of the areas. The agency’s plans to implement the recommendations included steps to develop a comprehensive communication plan, planned coordination between all relevant business and IT units to review and approve state requests for federal IT funds, and continued collaboration with states to test their system functionality. CMS offered various types of assistance to states transitioning to a different marketplace IT platform, including to the two selected states that transitioned to the federal platform—Hawaii and Oregon. Among other things, CMS’s assistance included periodic reviews of these two states’ transition plans and weekly calls to prioritize and address transition issues. CMS also issued a regulation to all states that included requirements regarding states’ transition plans and their coordination with CMS and HHS; however, this regulation was not finalized until after Hawaii and Oregon had initiated their transition efforts. States’ officials reported costs of approximately $84.3 million, collectively, to transition to the federal platform. The two states’ transition efforts included making changes to their Medicaid systems and the states mainly relied on Medicaid matching funds from CMS to do this. Further, the two states encountered challenges during their transitions. These challenges were related to accelerated transition time frames, difficulties reassigning functional marketplace responsibilities, delays in receiving approvals and decisions needed from CMS, and trouble accessing historical consumer data in previous marketplace IT systems developed by vendors. Among the assistance that CMS provided for states that sought to transition to a different marketplace IT platform was conducting periodic oversight activities. Specifically, for all state-based marketplaces, the agency conducted oversight through the Open Enrollment Readiness Reviews, as well as through sustainability consultations and weekly telephone calls during which CMS reviewed states’ plans and milestones. The Open Enrollment Readiness Review process involved CMS reviewing each state-based marketplace’s readiness for the upcoming open enrollment period by discussing topics such as IT system and business function readiness, application processing and notices, inconsistent data, enrollment transactions, and previous reviews’ findings. After the discussions, follow-up written communication from CMS was sent to the states highlighting outstanding action items that needed to be completed. In addition, CMS sustainability consultations evaluated state marketplaces’ self-sustainability in the absence of federal funding from marketplace establishment grants. These sustainability evaluations included reviews of the state marketplaces’ revenue sources, cash reserves, management team stability, and Medicaid and marketplace systems’ integration. Lastly, weekly telephone calls with states included discussions of the states’ plans and milestones associated with IT releases and testing, identification of technical issues, and mitigation and contingency plans. Further, in March 2016, subsequent to Hawaii and Oregon’s transition to the federal platform, CMS issued a regulation that set out more defined transition guidance. The regulation, for the first time, provided generally applicable guidance about the transition process and associated requirements. It required state marketplaces that seek to utilize the federal IT platform for selected functions to, among other things, coordinate with CMS, including joint development of a transition plan, which is to consist of a project plan with proposed milestones. CMS officials are to review the plans and help the states prioritize their planned transition milestone dates. Also, in October 2016, CMS finalized an agreement for states seeking to transition to the federal IT platform. That agreement became effective on November 1, 2016, with 1-year renewal options. The agreement established the mutual obligations and responsibilities of the transitioning states and CMS in areas including eligibility and enrollment, maintenance of related IT systems, call center operations, and casework support. However, the issuance of the regulations and agreement discussed above occurred after Hawaii and Oregon had already begun their April 2014 and June 2015 transitions to the federal platform. Thus, the requirements and documented guidance included in the regulation and the agreement were not available to these states when they initially undertook their transition efforts. Hawaii and Oregon officials described the assistance that CMS provided to their states, as follows: Hawaii began its year-long transition in June 2015 from a state-based marketplace that was operated by the Hawaii Health Connector to a state-run, state-based marketplace using the federal platform. The state completed an additional transition to be a federally facilitated marketplace in January 2017. At the time that Hawaii carried out its transition in 2015, CMS had not yet provided the states with documented guidance; however, CMS officials provided milestones to Hawaii that needed to be completed to fully transition to the federal platform. These milestones included dates when the marketplace plan and issuer data should be submitted and transferred, Medicaid account transfer functionality should be completed and tested, and notices and other related enrollee communications should be completed. In addition, Hawaii officials said that, throughout the transition, they participated in weekly telephone calls with CMS subject matter experts to discuss requirements such as consumer outreach and the interface development needed for account transfers. Oregon began its transition in April 2014 from a state-based marketplace operated by the non-profit organization, Cover Oregon, to a state-run marketplace using the federal platform, under the state’s Department of Consumer and Business Services. According to Oregon officials, CMS did not have guidance available at the time that the state decided to transition. However, state officials said they participated in many phone calls throughout the transition with CMS officials within CCIIO and CMCS to identify whether the state marketplace, state Medicaid, or CMS would be responsible for certain marketplace functions after the transition from Cover Oregon to the Department of Consumer and Business Services. The total reported IT costs associated with Hawaii’s and Oregon’s transition to using the federal platform were approximately $84.3 million, collectively. State officials said that the transition costs were incurred to make changes to their Medicaid systems and connect their systems to the federal IT platform. Hawaii and Oregon officials told us that the two states primarily relied on federal Medicaid matching funds to cover the cost of their transition efforts. Hawaii officials reported to us that their total IT costs associated with the initial transition to using the federal platform, begun in June 2015, were approximately $27.0 million as of June 2016. According to the officials, the transition costs primarily covered the development of functionality to transfer account files between Hawaii’s existing state Medicaid system and the federal platform. These included expenses for staff to carry out project management, technical assistance, and independent verification and validation activities. In addition, the costs covered funding for staff to perform design, development, and implementation work in order to enable the system to determine minimum essential coverage. The officials stated that these transition costs were largely funded by the federal government through Medicaid 90/10 and 75/25 matching funds. With regard to the transition to become a federally facilitated marketplace in June 2016, state officials said the primary reason Hawaii decided to do so was because the state lacked additional federal grant funds and the state legislature had denied subsequent requests for the additional funding for marketplace development and operations. The officials said that, in November 2016 they determined that, since the state was already relying on the federal platform for the previous plan year, the only IT work required to be completed would be a small update to their website. Thus, there was no cost associated with the second transition in January 2017. Oregon officials reported to us that their total IT transition costs to the federal platform, begun in April 2014, were approximately $57.3 million, as of November 2016. According to the officials, the transition costs primarily covered the modernization of the state’s legacy Medicaid system through the use of shared computer software code from the Kentucky marketplace system and the development of functionality to transfer account files between its Medicaid system and the federal platform. Because Oregon’s previous state-based marketplace IT solution was intended to integrate and modernize the state’s legacy healthcare systems, the decision to switch to the federal IT solution left the state still needing to modernize its legacy Medicaid system. In addition, the transition effort included expenses for staffing, professional services, computer hardware, and the software and service fees to host the new Medicaid system. The new Oregon Medicaid system went live in December 2015. Oregon officials reported to us that of the $57.3 million in IT costs, approximately $56.6 million were largely funded by the federal government through Medicaid 90/10 and 75/25 matching funds and approximately $662,000 were funded by marketplace grants from April 2014 to November 2014. The latter amount was needed to enable Oregon residents to apply for coverage through the federal system, HealthCare.gov, for plan year 2015. Hawaii’s and Oregon’s transition time frames and IT-related costs are summarized in table 2. While the two selected state-based marketplaces successfully transitioned to the federal platform, officials from each state identified a number of challenges they encountered in doing so. Hawaii officials identified challenges that stemmed from difficulties dealing with: Accelerated transition time frames and reassigned marketplace responsibilities. According to Hawaii officials, the state had originally planned to transition to the federal platform in October 2016 but then accelerated its efforts to address the unexpected shutdown of the Hawaii Health Connector in December 2015, due to financial problems. The accelerated and abrupt transition forced the state’s officials into the difficult task of having to find a way to continue operations and provide consumer support almost a year ahead of their original plan, since that plan did not have the Hawaii Health Connector transferring its marketplace responsibilities to state officials until October 2016. The unexpected shutdown also meant that the state had to quickly reassign functional marketplace responsibilities among seven different state departments, which made it more complicated to coordinate and find funding to continue marketplace- related operations to support the open enrollment for plan year 2016. Additionally, the reassignment of responsibilities was complicated by the fact that the Hawaii First Circuit Court appointed a receiver to understand all of Hawaii Health Connector’s obligations, dissolve its assets, and settle the organization’s financial dealings with creditors and debtors. Delays in communications with CMS. Hawaii officials stated that the distance and time differences between CMS officials working in the Eastern time zone and Hawaii officials working in the Hawaii- Aleutian time zone caused delays in communications between these officials. State officials said the geographical distance and resulting time difference of 6 hours in working with CMS officials primarily based in the District of Columbia and Maryland caused delays in receiving marketplace related information from CMS, such as notifications about when service disruptions would occur. However, Hawaii officials noted that, for the most part, this challenge subsequently was overcome by improved communications with the agency’s officials, thereby resolving former marketplace service disruption issues, such as planned federal platform system outages and automatic re-enrollment of individuals. Oregon officials identified challenges that stemmed from difficulties dealing with: Transitioning without the benefit of a transition guide to follow. According to Oregon officials, since their state was the first to transition to the federal platform, there was no other state model or guide to follow. In addition, CMS at the time, did not have documented guidance and requirements for states that wanted to transition to a different marketplace type. This required Oregon officials to spend many hours in discussions with CMS officials to figure out what functions CMS and the state each would be responsible for performing. Delays in receiving approvals from CMS. According to Oregon officials, CMS did not always make decisions related to Oregon’s transition in a timely manner. Oregon officials noted that there were delays with CMS issuing the approvals needed to allow the state’s residents to create accounts with HealthCare.gov before open enrollment started in November 2014. The officials said that being able to create those accounts in advance would have made enrollment easier for Oregon residents. In addition, the federal call center scripts initially were not customized with the HealthCare.gov system, so the responses Oregon residents received from HealthCare.gov were not always clear or accurate, according to the officials. CMS officials within CCIIO and CMCS stated that, at the time when Oregon began discussions with the agency about transitioning to the federal platform, CMS was also addressing priorities from the 2014 open enrollment and needed to balance resources spent on Oregon’s transition with other IT development and enhancement activities that were needed. Correcting errors with account transfers in the Medicaid system. Oregon officials noted that, in December 2015, initial account transfers between Oregon and CMS encountered errors due to a configuration issue related to outdated CMS guidance. Oregon had to make changes to its eligibility system in order to process account transfers because the actual transfers varied from CMS’s previously published specifications in certain areas. Meeting CMS’s 60-day notice for technical changes. State officials said CMS’s practice of providing approximately 60 days’ notice for technical changes to account transfer specifications was challenging to comply with in a timely manner due to the length of time that was needed for Oregon’s system integrator to make the requisite system changes. As noted above, these state transition challenges stemmed from a variety of issues including compressed time frames, split priorities, and unclear marketplace responsibilities. Some of these challenges may have been alleviated if the states had the benefit of documented transition-related guidance that exists now, available to them when they first initiated their transitions. In addition to the aforementioned challenges, for both selected states, a common challenge involved accessing the historical marketplace IT systems containing consumer data that had been developed by the initial contractors for Cover Oregon and the Hawaii Health Connector. In Hawaii, the court-appointed receiver for the state said the Hawaii Health Connector system integrator contractor had possession of the previous Hawaii marketplace IT system that contained the consumer data. As of November 2016, Hawaii and the contractor were still discussing the terms of compensation under that contract, and Hawaii officials did not know when those issues would be resolved. Oregon officials informed us in July 2016 that their state and its contractor were in litigation, and the contractor was storing Oregon’s previous marketplace IT system and the data within it in an archive. Oregon officials within the Department of Consumer and Business Services stated that they had a local archive of the data from the previous marketplace IT system, but did not have access to the actual contractor- based IT system after March 2016. With regard to this matter, CMS officials within CCIIO and CMCS noted that their continued work had included extensive discussions with both states on options and mitigation strategies to retain the data and maintain compliance with the marketplace 10-year archival requirement for consumer and enrollment data. The officials said that the historical data are needed so that states can maintain the ability to process actions that include marketplace eligibility appeals (which could span multiple years) and submission of enrollment data. The CMS officials also stated that they had continued to refine their operational processes and policy associated with transitioning to the federal platform, as well as provide guidance and regulatory requirements for states that transitioned their marketplaces to other models, with particular focus on those that transitioned to the federal platform. Further, the officials stated that they continued to work with Hawaii and Oregon to explore opportunities to enhance the federal platform’s functionality to better support the state and other state-based marketplaces on the federal platform, where feasible. They added that their continued work also included frequent communications with both states to ensure a smooth coverage transition for residents of each state. CMS had processes in place to assist the selected states’ with their efforts to financially sustain the development and operations of their marketplaces, including supporting IT systems. These processes included reviewing sustainability plans, reviewing annual independent financial audit reports, and conducting and responding to sustainability risk assessments. However, in providing its assistance, CMS did not always ensure that the four selected states’ sustainability plans and financial audit reports were complete, or that the states had complied with PPACA and CMS requirements regarding financial audit reporting. Additionally, CMS did not clearly define its risk assessment processes, as suggested by Standards for Internal Control. As previously noted, PPACA required state marketplaces to be self- sustaining as of January 1, 2015, and, in turn, CMS developed marketplace blueprint requirements to assist states with meeting the act’s requirements. As part of the blueprint for approval of a state-based marketplace, the agency required states to submit an operational budget and management plan for its oversight, and to include proposed budget information for the upcoming 5 years from the initial year of operations, and long-term strategies for financial sustainability. According to CMS officials within CCIIO, the states’ plans are intended to inform CMS regarding the state-based marketplaces’ sustainability. According to agency guidance, the plans are also used by CMS to assess and respond to marketplace sustainability risks. To aid in the reporting of the sustainability plans, CMS created a 5-year budget forecast template for states to complete and submit. The template called for high-level reporting of sustainability factors, to include marketplace enrollment, revenue, expenditures, reserves, and marketplace status over 5 years. In addition, contractual spending and questions related to ongoing IT costs were to be included in the plans. The 5-year span of budgets and enrollments that were to be reported in the plans included current-year projections and actuals, as well as forecast projections for the forthcoming 4 years. Sustainability guidance, dated October 2016, further required states to update and submit their sustainability plans to CMS twice a year. While CMS received complete 2016 sustainability plans from New York and Oregon, it did not ensure that it had complete sustainability plans with the full 5-year budgets from Hawaii and Minnesota. Specifically, Minnesota and Hawaii did not include the entire 4 years of budget forecasts in their plans. Hawaii’s sustainability plan dated May 2016 only contained the 2016 budget and enrollment amounts, but forecasts for the forthcoming 4 years were missing. Instead of forecasted amounts, Minnesota’s 2016 sustainability plan used duplicated budget and enrollment amounts, in which certain budget numbers, such as equipment and supply costs, were the same from 2016 to 2019. Hawaii, Minnesota, and New York state officials said that CMS did not provide them policies or procedures to help guide state marketplaces on aspects of sustainability planning, such as budget and enrollment forecasts. Minnesota and Oregon officials also reported that completing 5-year budgets was difficult due to problems with forecasting, and that budgets beyond the regular 3- and 2-year state budgetary cycles, respectively, were difficult to formulate. CMS officials further stated that Hawaii’s transition to a federally facilitated marketplace for plan year 2017 had precluded any future budgets or enrollment forecasts. CMS officials within CCIIO and CMCS stated that they communicate with states to resolve issues in providing complete budget forecasts. The officials acknowledged that states face uncertainties because the marketplaces are new programs, and stated that they are considering whether they should ask for 3-year budgets verses 5-year budgets. While asking states for a 3-year budget instead of a 5-year budget may be less of a burden on states to provide complete budgets, the smaller timeframe may not fully inform CMS oversight of the long-term financial sustainability for marketplaces, which are new systems that face multiple uncertainties. Further, if CMS does not take steps to ensure that states provide sustainability plans with complete 5-year budget forecasts, per its guidance, then CMS may not be fully informed of the state-based marketplaces’ sustainability factors. Incomplete sustainability plans may also limit the agency’s ability to assess and respond to state marketplace sustainability risks. PPACA and HHS regulations require state-based marketplaces to provide an annual independent financial audit report, to include activities, receipts, and expenditures, for CMS’s oversight. Also, submission of the independent financial audit report is a requirement of CMS’s annual reporting SMART process. HHS regulations stipulate that audit reports, in addition to accounting for receipts and expenditures, should follow generally accepted government auditing standards (GAGAS). According to GAGAS, audit reports should include information such as a review of compliance, internal controls, and related financial policies and procedures. Further, according to CMS guidance, the independent financial audit reports are to provide CMS with insight into marketplace IT self-sustainability efforts and compliance, since IT costs are a large part of the marketplace budgets. This guidance calls for the agency to use the reports to inform sustainability risk assessments and responses provided to the states. While CMS took steps to collect and review marketplace financial audit reports, it did not ensure that the four selected states always provided audit reports or that the reports were complete. CMS reviewed New York’s and Oregon’s relevant financial audit reports for 2015, which included the information that GAGAS required, such as information on compliance and internal controls. However, CMS did not ensure that Hawaii provided a financial audit report for 2015. In addition, although Minnesota submitted a financial audit report in 2015, CMS did not ensure that the report included all necessary information. For example, the Minnesota report was not specific to the marketplace business operation or its IT platform and, instead, included financial activities for all state programs and activities. While marketplace receipts and expenditures were included, a review of marketplace compliance, internal controls, and financial policies and procedures were not. Since these reviews were not included in Minnesota’s submission of its financial audit report to CMS that year, CMS did not have visibility into these aspects of the state’s marketplace- specific activities or complete insight into its sustainability efforts. In discussing their reporting, Hawaii officials in the Office of Community Services and Department of Human Services said they are not required to develop a financial audit report now that the state has switched over to being a federally facilitated marketplace. Further, according to the state’s officials, a court ruled that an independent financial audit of the defunct Hawaii Health Connector would be unfeasible and impractical since the Connector’s records are not amenable to audit, nor is there a Connector official available to sign off on the audit. A Minnesota marketplace official responsible for compliance and program integrity stated that MNsure was in compliance with PPACA and HHS regulations requiring the submission of financial audit reports, since its report included marketplace financial statements. In addition, the official stated that the cost to complete an independent financial audit specific to the marketplace would be an undue burden. The official added that CMS had accepted the overall state financial audit report as sufficient. CMS officials stated that the agency accepted the overall Minnesota state audit report given Minnesota’s challenges with reporting and said they were working with the state to mitigate challenges in providing the required audit report. Further, these officials stated that relevant audit information may be gathered through alternate channels and that they do not plan to enforce Minnesota’s compliance in providing an independent financial audit report specific to the marketplace in the short term. CCIIO officials further stated that, while they can communicate a lack of compliance to the states, they are statutorily limited in regard to enforcement mechanisms for state marketplaces. The officials added that, in the long term, they hope to address reporting limitations through providing additional guidance to the states. Nevertheless, although CMS took steps to ensure that the states submitted annual financial audit reports, the agency did not ensure the states followed regulations and guidance, which decreased CMS’s visibility into state marketplace sustainability and could increase sustainability risks. While individual states may have unique situations that preclude the submission of complete independent financial audit reports, the law and guidance are clear that states must submit annual financial audit reports that follow GAGAS requirements, including a review of compliance, internal controls, and related financial policies and procedures. Further, although CMS officials stated that alternate channels may be used to gather audit information, the agency’s guidance specifically refers to the annual financial audit reports as one of the primary sources for evaluating state marketplace sustainability. If CMS does not ensure that states provide complete annual financial audit reports, it may not have visibility into marketplace IT-related financial activities such as receipts, expenditures, internal controls, and financial policies and procedures. A lack of relevant financial audit reports can lead to uninformed CMS sustainability risk assessments and responses, which could increase state marketplace sustainability risks. Standards for Internal Control calls for agencies to assess and respond to risk using clearly defined measurable terms, objectives, and risk tolerances, to include a clear categorization process, reliable information, and a clear response to risks. Clearly defined objectives state what is to be achieved in specific, measurable terms, as well as how the objectives will be achieved, who will achieve them, and in what time frames. In addition, clearly defined risk tolerances set the acceptable level of variation in performance relative to the objective. These risks should be assessed using relevant data from reliable sources in a timely manner based on identified information requirements. According to CMS policy, these required information sources include state sustainability plans and independent financial audit reports. Additionally, agencies should respond to risk based on the significance of the risk. According to CMS guidance, these processes address issues that may impact state marketplace IT required financial self-sustainability. In order to assess and respond to risk, CMS had established processes for the following activities for all state-based marketplaces, including those that use the federal platform: collects sustainability information. CMS uses financial audit reports and sustainability plans to create sustainability risk assessments and conduct consults for each state-based marketplace. assesses state marketplace risk. The agency conducts risk assessments by reviewing factors such as marketplace IT costs, functionality, and operations stability. It then scores states’ marketplace self-sustainability risk factors, and categorizes marketplaces based on a risk rating of low, medium, medium-high, or high. Low risks indicate a stable IT infrastructure and budget while higher risks indicate challenges to marketplace sustainability, such as insufficient projected revenues, insurance issuer volatility, or IT system challenges. responds to assessed risk. CMS responds to identified marketplace sustainability risk, including IT system sustainability risks, through sustainability consults. The agency discusses marketplace sustainability with the state, addressing specific issues that may impact sustainability, and identify areas for technical assistance. Agency officials said they also provide assistance as needed through phone calls with state officials. follows up with the state. CMS provides the results of consults to the states in a site visit report. This report conveys action items concerning sustainability compliance and/or recommendations for self-sustainability based on industry best practices. In addition, according to CCIIO officials, marketplaces assessed at medium-high or high risk receive more intensive assistance from the agency. While CMS took steps to establish a sustainability risk assessment process, there were numerous shortcomings with the agency’s implementation of that process for the four selected states. Specifically, the agency’s sustainability risk assessment and consult activities were not based on a fully defined risk process, to include having fully defined measurable terms, a clear categorization process, reliable information, and a clear response to risk. CMS’s procedures outlined the steps in the agency’s sustainability risk assessment process, but the procedures did not always define risk factors in clear and measurable terms. For example, the procedures did not define the risk factor for enrollment target tolerances in quantifiable terms. Instead, the agency used the terminology “close to expected” or “lower than expected.” Additionally, while the agency assigned the four selected state marketplaces in categories of related sustainability risk, there was no clear categorization process defining how risk assessment scores of low, medium, medium- high, or high risk were obtained. Specifically, CMS did not provide documentation of defined score thresholds, such as what score out of the maximum determined the sustainability risk categorization. It also did not define a consistent baseline for risk scores in its documentation or assessments—we identified three different baselines in CMS’s risk assessments of the four selected states. As an example, we found that CMS assessed one state based on a scale of 59 points of weighted risk factors, while other states were assessed on scales of 67 or 76 points. In addition, as mentioned previously, the sustainability plans and financial audits from the selected states that were used for the risk assessments were not always complete and, thus, were not always reliable sources of information. Further, CMS did not provide documentation defining a process for responding to assessed risks and its response to risk as documented in the site visit reports was not based on the significance of the risk. The site visit reports addressed no more than one risk factor per each of the four states we reviewed, despite the agency categorizing some of the selected states at a medium-high or high risk level, with multiple assessed risk factors. For example, in one state’s site visit report, CMS provided a recommendation for a lower weighted risk factor— marketplace reserves. However, the agency did not address other factors that it designated in the state’s associated risk assessment as being of highest risk, such as a limited revenue source or IT functionality shortcomings. Moreover, the agency’s policy and procedure for risk response did not vary based on risk, so the internal guidance for sustainability assistance is the same for a high-risk marketplace as a low- risk marketplace. CCIIO officials said that risk assessments were never meant to be quantitative and that they used their best judgment after looking at a number of areas to rate marketplace risks. Further, the officials said they did not tailor responses to different levels of assessed risk ratings because the assessments were considered an internal guide to provide CMS leadership with a general idea of a marketplace’s sustainability risk and were used as a tool to determine what technical assistance states needed. However, if CMS does not take steps to define sustainability objectives in measurable terms, to include a clear marketplace risk categorization process, use of relevant data from reliable sources, and responses based on risk significance, then it is possible that risks will not be correctly assessed and responded to by the agency. Accordingly, if marketplace sustainability is not correctly assessed and responded to, CMS may not be able to assist states in achieving their required financial self- sustainability. Among other things, leading practices emphasize the importance of having performance metrics and developing performance plans to identify the most important metrics to guide decisions and measure IT performance. In addition, CMS’s guidance calls for it to ensure that states have documented performance measurement plans and to conduct operational analysis reviews to examine the operating status of state marketplace IT systems using key performance indicators. Also, during the open enrollment period, CMS requires states to submit a report of weekly performance indicators, which includes some metrics related to the operational performance of marketplace systems, such as the number of applications completed electronically, the total number of website visits, and website offline time, among others. However, CMS did not consistently monitor the performance of IT systems for Minnesota and New York—the two selected states that operated state-based marketplace systems. Specifically, CMS did not ensure that the two states had developed, updated, and followed performance measurement plans. In addition, it did not conduct reviews to analyze the operational performance of the selected states’ marketplace IT systems against an established set of performance parameters to evaluate whether the states were performing in an efficient and effective manner. As for IT metrics that were collected from the states, CMS did not link these metrics to performance measurement goals or establish targets for performance. In applying its Enterprise Life Cycle framework to monitor states’ marketplace IT systems, CMS is to ensure, as part of its operational readiness review, that states have documented performance measurement plans. These plans are to be used to assess the business value of states’ marketplace IT. Further, the Enterprise Life Cycle framework requires that states evaluate performance metrics and share their results with responsible parties, such as federal officials and state project and business managers. CMS also provides states a template that they can use, which includes a section on project measurement objectives, performance metrics, and thresholds, which set parameters for target performance. In addition, our previous work has emphasized the importance of performance metrics to assess the actual results, effects, or impact of a program or activity compared to its intended purpose. We also emphasized that leading practices include the development of plans to identify the most important metrics to guide decisions, and document goals and metrics to measure IT performance. Our previous work noted that the performance measurement approach should be holistic, or seen in terms of the operation as a whole, in order to identify a comprehensive suite of metrics. Further, our work has stated that the performance measurement approach should be continuously assessed and followed by regularly reviewing metrics, goals, and targets; and adjusting these as necessary. CMS reviewed the two selected states’ progress on marketplace IT projects, but had not ensured that these states documented, updated, and followed their performance measurement plans to demonstrate that they had identified and selected the most important metrics to guide decisions. In addition, CMS had not ensured the selected states continuously assessed and adjusted performance metrics and targets as appropriate. Minnesota: CMS did not ensure that Minnesota updated and followed its performance measurement plan to show that the state had continuously assessed its performance measurement approach and adjusted metrics and targets as necessary. Specifically, in the September 2013 operational readiness review, CMS noted that Minnesota officials had partially identified performance metrics in their project planning documentation, and had partially evaluated performance metrics and shared results with responsible parties. CMS also noted that the state had not developed or identified its operational metrics. Subsequently, to address CMS’s observations, in December 2013, Minnesota drafted a performance measurement plan to identify performance measurement goals and targets. The December 2013 plan included technology goals such as ensuring that at least 90 percent of users have real-time, online access to the marketplace website and decreasing code defects, or software errors, per release by at least 60 percent. Information in the plan stated that it was to be reviewed quarterly and updated as needed. However, in June 2016, Minnesota officials from the state’s IT Services organization said that the 2013 performance measurement plan was likely developed by a contractor and had not since been updated or followed. According to the officials, while the state did not follow its performance measurement plan, the state was monitoring marketplace IT metrics related to its technology goals, such as system availability and unexpected down time. In addition, Minnesota developed a service-level agreement that included target metrics for system availability, although it did not require the reporting of defects. Neither CMS nor state officials provided evidence that the state was actively monitoring these metrics related to their technology goals. New York: CMS had not ensured that New York developed a performance measurement plan. In August 2013, CMS provided New York a satisfactory rating for evaluating performance metrics and sharing results with responsible parties; however, it noted that the state had only partially documented performance metrics in a plan. During this review, CMS also stated that New York had not developed or identified operational metrics. Although New York did not have a performance measurement plan, according to state officials, performance metrics were documented in its June 2016 oversight and monitoring plan. This plan included metrics for the contractor responsible for New York’s marketplace IT operations. Information in the oversight and monitoring plan referred to IT metrics such as system downtime, timeliness of file processing, real-time transaction processing, backup and recovery implementation, failover and fallback capability, and disaster recovery infrastructure. In addition, New York developed a service-level agreement that included targets. For example, the service-level agreement required that New York’s marketplace system be available at least 98.5 percent of each month. However, the oversight and monitoring plan and service-level agreement did not include IT performance goals or related metrics. For example, the oversight and monitoring plan did not include metrics that New York IT officials said they use to monitor the performance of marketplace IT systems, such as metrics related to defect creation and remediation, the number of batch jobs, electronic data interface files that were rejected, and notices and related backlogs. These metrics were not documented or tied to performance goals or targets in a plan. In addition, according to New York marketplace officials, the goal of the IT service-level agreement metrics for its contractor is to ensure that consumers can readily and easily access the health insurance application and systems needed to enroll in coverage. However, these stated goals were not documented in the oversight and monitoring plan or clearly linked to the IT performance metrics. CMS officials from OTS said states were expected to address the issues that it identified in its reviews, such as partially completed performance measurement plans, and that the agency monitors marketplace system performance through daily calls with states. The officials said that Minnesota had not updated its performance measurement plan as requested and that it falls upon the state to ensure that service level agreement metrics are met, and if not met, that corrective actions will be taken. In addition, the officials stated that, instead of a performance measurement plan, the agency reviewed New York’s IT contractor service level agreements, which represented the state’s operational metrics. However, CMS did not ensure that these metrics were tied to performance goals in a plan in accordance with leading practices. Further, according to CMS officials, the Enterprise Life Cycle required project measurement plans in the planning and design phase, but not in the operations and maintenance phase, and operational states were not required to submit updated performance measurement plans. However, while CMS’s Enterprise Life Cycle guidance did not explicitly require states to update performance measurement plans during operations and maintenance, leading practices stress that it is important to have updated performance measurement plans to properly assess effectiveness in meeting stated operational goals. Because CMS had not ensured that states documented, updated, and followed performance measurement plans for their state marketplace systems, the agency did not have the assurance that states had taken a holistic approach to developing performance metrics to assess actual results as compared to intended goals. Further, without reviewing states’ plans, CMS could not ensure that states had carefully identified and selected the most important metrics to guide decision making and organizational operations, or that states continuously assessed and adjusted performance metrics and targets as appropriate. As part of the Enterprise Life Cycle framework, to monitor system performance, CMS and states are required to conduct an operational analysis review to examine the operating status of the marketplace system through a variety of key performance indicators and determine whether the system is performing in an efficient and effective manner. In addition, according to leading practices, operational analysis is a key management tool to examine the performance of an operational initiative and measure that performance against an established set of performance parameters. The operational analysis should consider how objectives could be better met and how costs could be saved. CMS had not conducted operational analysis reviews for states, including Minnesota and New York. Instead of conducting operational analysis reviews, the agency developed OTS reports in November 2015 regarding selected states’ systems to prepare for the 2016 open enrollment period. The reports included discussions of these states’ system performance. For example, the OTS report for Minnesota included a summary of the state’s system functionality and performance. Specifically, the report noted that Minnesota discovered quality issues with the software used for eligibility determinations and put a plan in place to remedy quality issues. In addition, the report noted that Minnesota was using scenarios to test site performance and assessing whether the system had sufficient capacity to meet business needs. For New York, the OTS report included a summary of the state’s system functionality, system performance, and stability of application bandwidth to handle peak volumes. It discussed business value, such as the number of eligibility determinations (2.7 million eligibility determinations, and over 78.9 percent enrolled in a health care plan). The report also identified risks for the open enrollment and estimated operating costs for the system. Nevertheless, while these reports discussed metrics on New York’s performance, such as the number of eligibility determinations, and Minnesota’s consideration of performance issues, they did not include key performance indicators to show whether the states were performing in an efficient and effective manner. Specifically, the reports did not include metrics or targets that might define performance success. For example, the reports did not discuss metrics related to application processing time or application backlogs in the marketplace IT system. CMS officials from OTS and CCIIO said that they had not conducted operational analysis reviews because they conduct open enrollment readiness reviews instead. The officials also said that states were still developing their systems for the 2017 enrollment period. Additionally, CMS officials said that they had not established key performance indicators because states are responsible for developing such measures and adjusting performance, and that the key performance indicators would be in the states’ service-level agreements. The officials added that throughout the year, OTS meets with states biweekly to track their progress on their software releases, and discuss and assist with resolution of any issues they may be encountering. However, the open enrollment readiness review included operational areas such as IT system and business functions, but did not note discussion of performance in terms of key performance indicators or other elements of operational analysis, such as how objectives could be better met or costs could be saved. In addition, while officials from the states in our review said they were still developing or updating certain aspects of their marketplace systems, their marketplaces were operational. Further, while service-level agreements and other metrics can be useful indicators, conducting an operational analysis is an opportunity to perform qualitative analysis of the utilization of technology in a holistic and strategic way in order to see where the states are relative to their performance indicators. Because CMS had not conducted operational analysis reviews to monitor the performance of the selected states’ marketplace IT systems in a systematic way, it had limited assurance that these states’ systems were performing in an effective and efficient manner. During the open enrollment period, CMS requires states to submit a report of weekly performance indicators. This report is to include metrics related to the operational performance of marketplace systems. For example, CMS requests that states include: the number of applications completed electronically and on paper In addition, according to our prior work, performance metrics should be linked to the program’s IT performance measurement goals and define what is important to the organization and what the baseline and target performance should be in order to determine how efficiently and effectively the systems are performing. Metrics can be used for an organization to define success, structure improvement efforts and identify early warning indicators of problems. CMS collected IT metrics from the two selected states that operated marketplace systems in the open enrollment weekly indicator template. The template included metrics related to marketplace systems’ operational performance, such as the number of applications completed electronically and the total number of website visits, among other metrics. However, the metrics were not clearly linked to performance measurement goals and did not include baselines and targets to indicate how effectively they are performing. Specifically, CMS did not define performance measurement goals, such as timely processing of applications, and therefore was unable to link metrics to those goals. In addition, CMS did not set, nor require states to set, performance baselines and targets for metrics such as the number of applications completed by electronic means or on paper, or the duration of times when the website was offline. According to CCIIO officials, the agency collects data to effectively and consistently monitor state-based marketplace performance and to identify any barriers to eligibility and enrollment in the marketplaces. The data are used to inform programmatic understanding of operations, but are not collected to diagnose or interpret IT system performance. According to these officials, because states have different internal goals, CMS assists states to oversee their own marketplace IT systems based on each state’s need, goals, and resources. Because CMS and states did not clearly link metrics to their own performance measurement goals and did not include baselines or targets for the marketplaces, the agency is limited in its ability to monitor whether states’ systems are performing efficiently and effectively. In addition, without baselines or targets, the agency may not have data to accurately monitor progress for continual improvement. Through periodic oversight and guidance, CMS offered assistance to states we selected for review that sought to transition their health insurance marketplaces to the federal IT platform. This oversight included reviews of transition plans and milestones and weekly calls between the agency and state officials to discuss transition progress. However, CMS guidance for states transitioning to the federal platform was not documented and finalized until after two states had already initiated their transitions. Those two states primarily utilized federal Medicaid funding to make the associated changes to their Medicaid systems to connect with the federal IT platform. The states encountered challenges in their transitions in part because CMS had not issued its transition-related guidance until after these states had transitioned. CMS assisted selected states with their effort to financially sustain the development and operations of their marketplaces (including supporting IT systems) by reviewing sustainability plans; reviewing annual independent financial audit reports; and conducting sustainability risk assessments. However, CMS did not provide consistent oversight of the four selected states’ programs because the agency did not take steps to collect complete sustainability plans or financial audit reports from all of these selected states. In addition, CMS did not clearly define its sustainability risk assessment process to assist states. Until CMS addresses these issues, the agency’s assistance with, and assessments of states’ marketplace sustainability may not fully account for risks that could impact or interrupt state marketplace IT operations. CMS’s guidance includes steps to monitor the performance of state- based marketplace IT system operations, including the collection of related IT performance metrics such as electronic enrollments and website traffic volume. However, its oversight did not ensure selected states’ systems performance was monitored in a way that was consistent with its guidance to states and leading practices. Specifically, CMS did not ensure that two selected states (Minnesota and New York) had developed, updated, and followed performance measurement plans. In addition, the agency did not conduct operational reviews to determine if marketplace IT systems for the two selected states were operating in an efficient and effective manner; it also did not establish performance measurement goals or targets for certain metrics it collected from states. As a result, CMS has been limited in its ability to determine whether state marketplace IT systems are performing efficiently and effectively and to provide early warning of potential problems for the overall state marketplace IT systems’ service delivery to consumers. We are making the following six recommendations to the Secretary of Health and Human Services to direct the Administrator of the Centers for Medicare & Medicaid Services to take action. 1. The Administrator of CMS should take steps to ensure that state- based marketplace annual sustainability plans, to the extent possible, have complete 5-year budget forecasts. (Recommendation 1) 2. The Administrator of CMS should take steps to ensure that all state- based marketplaces provide required annual financial audit reports which are in accordance with generally accepted government auditing standards. (Recommendation 2) 3. The Administrator of CMS should take steps to ensure that marketplace IT self-sustainability risk assessments are based on fully defined measurable terms, a clear categorization process, and a defined response to high risks. (Recommendation 3) 4. The Administrator of CMS should take steps to ensure that states develop, update, and follow performance measurement plans that allow the states to continuously identify and assess the most important IT metrics for their state marketplaces. (Recommendation 4) 5. The Administrator of CMS should take steps to conduct operational analysis reviews and systematically monitor the performance of states’ marketplace IT systems using key performance indicators. (Recommendation 5) 6. The Administrator of CMS should take steps to ensure that metrics collected from states to monitor marketplaces’ operational performance link to performance goals and include baselines and targets to monitor progress. (Recommendation 6) We provided a draft of this report to HHS for comment. In its written comments (reproduced in appendix II), the department concurred with two of our recommendations, partially concurred with two recommendations, and did not concur with two recommendations. HHS concurred with our second and third recommendations which, respectively, called for CMS to ensure that all state-based marketplaces provide required annual financial audit reports that are in accordance with generally accepted government auditing standards, and ensure that the marketplace IT self-sustainability risk assessments are based on fully defined measurable terms, a clear categorization process, and a defined response to high risks. HHS stated that it will continue to provide technical assistance to state marketplaces regarding independent financial audits. The department also stated that it will refine its marketplace self-sustainability risk assessment processes to provide greater insight into the state marketplace sustainability efforts and to identify areas where states may need assistance. Taking steps to provide technical assistance to the states is important and CMS’s efforts to refine the risk assessment processes can provide greater insight into the state marketplace sustainability efforts and areas of needed assistance. HHS partially concurred with our fourth recommendation that CMS ensure that states develop, update, and follow performance measurement plans that allow the states to continuously identify and assess the most important IT metrics for their state marketplaces. While HHS did not specifically identify which aspects of our recommendation it concurred with and which it did not concur with, the department stated that, as part of its Enterprise Life Cycle framework, state marketplaces were required to submit performance measurement plans during the planning and design phases. The department also stated that it will continue to monitor the state marketplaces’ IT metrics in the implementation phase. For the state marketplaces that are in the operations and maintenance phase, it stated that each marketplace is accountable for managing and reporting its own IT metrics in accordance with federal and state law. The department also emphasized its consideration of states’ variations in marketplace systems and reporting capabilities and the associated burden of reporting IT metrics. However, as we noted in our report, CMS and certain states were not always able to provide evidence of performance measurement plans that were in accordance with the agency’s policy, nor was evidence always provided that these states updated and followed their performance measurement plans according to best practices. While CMS required submission of performance measurement plans in the planning and design phases, best practices state that performance measurement plans should be continuously updated and followed. Without ensuring that the states documented, updated, and followed performance measurement plans, CMS may not have reasonable assurance that the states established IT metrics to assess their results compared to their intended goals for their marketplace systems. HHS also partially concurred with our sixth recommendation that CMS ensure that metrics collected from the states to monitor marketplaces’ operational performance link to performance goals and include baselines and targets to monitor progress. HHS did not specifically identify which aspects of our recommendation it concurred with and did not concur with; however, the department stated that, while it requests performance measures from the state marketplaces, once the marketplaces are operational, states are responsible for monitoring their own performance measures. HHS also stated that it will continue to review IT metrics of state marketplaces in the implementation phase of their systems, but emphasized the burden on states and variations in state system reporting capabilities. However, as we noted in our report, CMS did not ensure that the metrics it is collecting from the states are linked to performance goals as suggested by best practices. Without this linkage, the agency may continue to be limited in its ability to monitor whether the state systems are performing efficiently and effectively. Additionally, CMS may miss the opportunity to refine its current IT metrics collection to better balance its need for visibility into states’ performance without unnecessarily burdening states. HHS did not concur with our first recommendation that CMS ensure that state-based marketplace annual sustainability plans, to the extent possible, have complete 5-year budget forecasts. The department stated that it has updated its requirements, and is now requesting 2-year budget forecasts instead of 5-year budget forecasts. It also stated that this is part of a new streamlined and simplified process to collect timely, accurate, and relevant data while taking into consideration the burden on states and the variations in state budget cycles. However, CMS did not provide documented evidence of this process or justification for stating that a 5- year budget is not a reasonable time frame for sustainability planning. This also contradicts previous CMS blueprint guidance for state marketplace approval. While asking states for a 2-year budget instead of a 5-year budget may streamline the process and be less of a burden on states to provide complete budgets, the shorter time frame may not fully inform CMS oversight of the long-term financial sustainability and associated risks for marketplaces, which are new systems that face multiple uncertainties. Further, if CMS does not take steps to ensure that states provide sustainability plans with complete 5-year budget forecasts, per its 2016 sustainability guidance, then it may not be fully informed of the state- based marketplaces’ sustainability factors. Incomplete sustainability plans may also limit the agency’s ability to assess and respond to state marketplace sustainability risks. Thus, we continue to believe that CMS should ensure that state-based marketplace sustainability plans have, to the extent possible, complete 5-year budget forecasts. Lastly, HHS did not concur with our fifth recommendation that CMS conduct operational analysis reviews and systematically monitor the performance of states’ marketplace IT systems using key performance indicators. The department stated that it conducts Open Enrollment Readiness Reviews to assess marketplace key performance indicators, which, according to CMS officials, are similar to operational analysis reviews. However, as we noted in our report, Open Enrollment Readiness Reviews did not systematically report the key performance indicators or include discussions of other elements of operational analysis reviews, such as how objectives could be better met or costs could be saved. In not ensuring that these reviews include clearly identified key performance indicators, CMS may miss an opportunity to perform strategic analysis of the states’ utilization of their marketplace systems and it may continue to have limited assurance that these states’ systems are performing in an effective and efficient manner. Therefore, we continue to believe that CMS should conduct operational analysis reviews, as required by its guidance, to systematically monitor the performance of states’ marketplace IT systems using key performance indicators. HHS also provided technical comments, which we incorporated in the report as appropriate. We also provided relevant excerpts of this product to each of the four states included in our review—Hawaii, Minnesota, New York, and Oregon—and received responses, via e-mail or in writing, from all four states. In written comments, the State of Hawaii’s Department of Labor and Industrial Relations noted that the Hawaii Health Connector’s IT provider would be turning over consumer data to the state by the end of June 2017. The department also indicated that the sustainability budget discussed in our findings was prepared by the Hawaii Health Connector and not the state. The department further commented that the state submitted a sustainability budget in April 2016 that included complete forecasts through 2019, and that this budget was updated in May 2016. Further, it said that the updated budget only reflected 2016 as Hawaii planned to operate as a federally facilitated marketplace and no longer needed to plan for financial sustainability. We noted in our report that the Hawaii Health Connector’s IT provider plans to turn over consumer data to the state by the end of June 2017. We also recognize that the April 2016 sustainability budget had the required budget forecast. However, our discussion of the Hawaii sustainability budget referred to the more recent and updated sustainability budget that the state prepared in May 2016. With regard to Hawaii’s April 2016 budget, the budget was based on revenue assumptions, such as state funding from the legislature that had not been approved at the time of its submission and, thus, required revision once those uncertainties were resolved in May 2016. With regard to the applicability of our findings for the May 2016 budget, Hawaii did not officially begin the transition to a federally facilitated marketplace until June 2016, and was still operating as a state-based marketplace at the time the May 2016 budget was revised. In addition, as indicated by CMS’s 5-year budget template and sustainability guidance, the sustainability budget should include forecasted years. According to CMS officials, they use this data to inform its sustainability risk assessments. By not showing the total effect of certain assumptions or outcomes across all forecasted years, such as the decision of the state legislature to not fund marketplace operations, the marketplace missed an opportunity to demonstrate the total negative impact on the marketplace sustainably and associated budget numbers. In that way, a fully forecasted budget could have served as evidence of Hawaii’s justification for transitioning to a federally facilitated marketplace. The State of Hawaii Department of Labor and Industrial Relations’ comments are reprinted in appendix III. In written comments, the state of Minnesota’s MNsure marketplace noted that the organization continues to focus on making improvements related to accountability and transparency, including in areas such as federal and state audit reporting for the marketplace. Minnesota provided additional comments in which the marketplace noted that its 3-year budget sustainability plans show that the marketplace is and will be sustainable. While Minnesota’s marketplace may have policies that do not align with CMS’s requirements for projected budget time frames, it is nonetheless important that the states submit information as required to the agency so that CMS officials can perform oversight of states’ marketplace sustainability in a consistent manner. Further, the shorter time frame may not fully inform CMS oversight, including risk assessments and responses, of the long-term financial sustainability for marketplaces. MNsure also disagreed with our characterization of Minnesota’s independent financial audit as not specific to the state’s marketplace. The marketplace provided details, including statements of the sufficiency of its state financial audit report because it is in adherence with Minnesota’s state statutes and financial policies. Additionally, MNsure officials stated that alternate audits may detail relevant audit information. However, CMS’s guidance specifically requires states to develop an annual financial audit report specific to the marketplace as one of the primary sources for evaluating a state marketplace’s financial sustainability. As we stated in our report, if MNsure does not provide complete financial audits specific to the marketplace, CMS may not have the necessary transparency into marketplace IT-related financial activities such as receipts, expenditures, internal controls, and financial policies and procedures. MNsure’s comments are reprinted in appendix IV. In addition, technical comments provided by marketplace officials were incorporated into our final report as appropriate. In e-mail comments, the Executive Director of the New York State of Health marketplace disagreed with our conclusion that the New York marketplace did not have a performance measurement plan that directly tied to goals. The Executive Director stated that the state’s overall goals were to enroll New Yorkers in coverage and reduce the rate of uninsured persons in the state. However, these stated goals were not documented nor were they tied to any specific metrics, as noted in our report. Until New York’s marketplace has documented these goals in a performance measurement plan with clear ties to its metrics, CMS may not have visibility into New York’s marketplace performance metrics which state officials said guide their decision making and operations, or be able to ascertain that the state continuously assessed and adjusted performance metrics and targets as appropriate. The Executive Director also noted that the New York marketplace could not reconcile the amounts provided by CMS for their state. According to their records, the New York marketplace had spent or planned to spend $487.2 million in marketplace grants that included $182.8 million for IT costs. Additionally, the Executive Director stated that CMS had deobligated $64.8 million. However, the amounts provided by New York were as of April 2016. Our report included more recent data obtained from CMS in October 2016 that provided a consistent view of spending for the four states in our analysis. Other technical comments provided by marketplace officials were incorporated into our final report as appropriate. In e-mail comments, the Oregon Interim Administrator provided technical comments, which we incorporated as appropriate. As agreed with your offices, unless you publicly announce the contents of this report earlier, we plan no further distribution until 28 days from the report date. At that time, we will send copies to the Secretary of Health and Human Services and other interested parties. In addition, the report will be available at no charge on the GAO website at http://www.gao.gov. Should you or your staffs have questions about this report, please contact me at (202) 512-9286 or pownerd@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made major contributions to this report are listed in appendix V. Our objectives were to: (1) describe what actions the Centers for Medicare & Medicaid Services (CMS) has taken, if any, to assist states that have chosen to transition to a marketplace IT platform different from the one they originally used and identify the costs and challenges for states in making this transition; (2) assess what actions CMS has taken to assist selected states’ plans to ensure that the development and operations of marketplace IT systems can be financially self-sustained; and (3) assess the steps that CMS has taken to monitor the performance of the states’ marketplace IT systems. To address the objectives, we reviewed marketplace activities conducted by CMS and four selected states: Hawaii, Minnesota, New York, and Oregon. We selected these 4 states from the 17 states that operated their own marketplaces as of March 2016. To make the state selections, we considered four selection factors for the plan year 2016 enrollment period: total enrollment, total federal marketplace grant dollars, a previous GAO review, and whether or not the state transitioned its marketplace to the federal platform. Specifically, we first sorted states operating marketplaces by enrollment levels, from highest to lowest, based on plan year 2016 numbers reported by the Department of Health and Human Services (HHS). We then divided the states into four groups. Within each group, we sorted the states from highest to lowest by the total amount of federal marketplace grant funds awarded and selected the state with the highest amount of awarded grant funding. In two cases, we selected the state with the second highest federal marketplace grant award in the group because the states with the highest award levels had been included in a recent GAO review of state health insurance marketplace IT security and privacy. We also reviewed selected states to verify that the states used different systems integrator contractors. The selection resulted in two states that transitioned to using the federal platform (Hawaii and Oregon) and two that remained the same (Minnesota and New York). The four selected states were based on a nongeneralized sample and, thus, findings from our assessments of these states cannot be used to make inferences about to the full population of all state marketplaces. To assess the reliability of CMS’s data on state marketplace enrollment figures, we reviewed the agency’s data and interviewed state marketplace officials for the four selected states and asked how the state-reported enrollment figures were utilized in their sustainability plans. We determined that the data were sufficiently reliable for our purposes. To assess the reliability of CMS’s data on grant funds awarded and state- reported IT spending to establish, support, and connect to marketplaces, we assessed the reliability of the systems used to collect the information. We asked officials responsible for entering and reviewing the grants information in these systems a series of questions about the accuracy and reliability of the data. Among the sources of data used for our study, we reviewed a spreadsheet compiled by CMS’s Center for Consumer Information and Insurance Oversight (CCIIO) officials that contained state-reported grant funding data and marketplace IT project status information drawn from two separate information systems: CMS’s Grant Solutions and the Payment Management System. The spreadsheet was a consistent source of information that reflected the same cost factors for all states as of October 2016. Specifically, the spreadsheet tracked, among other things, the type and total amount of grant funding provided and available to each state, deobligated grant funding, as well as the time period for expending those funds. We also reviewed the data to determine if there were any outliers and other obvious errors in the data. For any anomalies in the data, we followed up with CMS officials to either understand or correct those anomalies. We determined that the data were sufficiently reliable for our purposes and noted any limitations in our report. While our report discusses state-reported IT spending based on CMS data, we did not verify the accuracy of the data states reported to CMS. To address the first objective, we obtained and analyzed CMS’s transition guidance that was distributed to assist all states. We also reviewed the actions CMS and states performed, such as communications and transition planning for the two selected states—Hawaii and Oregon—that transitioned from state-based to federal marketplace IT systems. To identify transition guidance and transition costs for the states, we also observed CMS’s and the selected state’s management tools, such as the Collaborative Application Lifecycle Tool, State-based Marketplace Annual Reporting Tool (SMART), Payment Management System, and Grant Solutions, for reporting and tracking of grant funding. In addition, we reviewed relevant CMS and state budget and grant documentation to determine associated transition costs. We also interviewed state marketplace officials within the two selected states to further identify transition costs and challenges faced during their transitions. We also interviewed CMS officials regarding identified challenges and their actions to assist the states in addressing them. To address the second objective, we reviewed the four selected states’ sustainability plans and CMS sustainability guidance provided to the states. To identify states’ plans for self-sustainability we reviewed development plans, financials audits, and grant documentation. To identify CMS sustainability guidance we reviewed the agency’s procedures for financial audit and sustainability plan collection, risk assessments, and sustainability consults. We compared CMS financial audit collection against applicable laws, regulation, and agency guidance. We also compared sustainability plan collection and risk assessments against leading practices. To understand how CMS monitors state sustainability, we observed its and the four selected states’ web-based management tools, such as SMART, Payment Management System, and Grant Solutions, for reporting and tracking of state marketplace self-sustainability. To determine the reliability of state sustainability plans, we reviewed the plans and relevant source data for anomalies and outliers, as well as related documentation including state audits and budgets. We also interviewed CMS and selected state officials regarding collection and processing of the data. We determined that the data in the sustainability plans were sufficiently reliable except where noted in our report. For the third objective, we reviewed CMS guidance provided to the state- based marketplaces which called for the monitoring and tracking of the performance of states marketplace IT systems. We identified steps CMS established for monitoring the performance of states’ IT systems. We compared the steps established by CMS to leading practices identified in our prior work and by the Office of Management and Budget. In addition, where available, we reviewed the two selected states’ system performance measurement plans and reports. This included Minnesota and New York—which operated state-based marketplace IT systems— and did not include Hawaii and Oregon—which relied on the federal marketplace IT platform operated by CMS and did not collect system performance metrics. We reviewed the use of tools such as CMS’s Collaborative Application Lifecycle Tool and Open Enrollment Weekly Indicators reports to facilitate the monitoring of state marketplace operations and performance. We analyzed whether CMS ensured that states followed leading practices for IT performance measurement guidance by assessing evidence of CMS and the two selected states’ marketplace performance measurement plans and reporting. To determine the reliability of state performance metrics reports, we reviewed the selected states’ reports for anomalies or missing data and conducted interviews with CMS and selected state officials regarding the collection and processing of the data. We determined that the data in the performance metrics reports were sufficiently reliable. We also assessed whether CMS had conducted operational analysis reviews of the two selected states’ marketplace IT systems using key performance indicators to determine whether states’ systems were performing in an efficient and effective manner. For all three objectives, we supplemented the information and knowledge obtained from our assessments of the program, project, and technical documentation by holding discussions with relevant CMS officials and interviews with state officials at selected state sites regarding their marketplaces. We conducted this performance audit from February 2016 to August 2017 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. In addition to the contact named above, Tammi Kalugdan (Assistant Director), David Hong (Analyst in Charge), Alexander Anderegg, Christopher Businsky, Debra Conner, Sandra George, Conor McPolin, Brian Palmer, Monica Perez-Nelson, Priscilla Smith, Merry Woo, and Elizabeth Wood made key contributions to this report.

Summary: The Patient Protection and Affordable Care Act required the establishment of health insurance exchanges—or marketplaces—to allow consumers to compare, select, and purchase health insurance plans. States can elect to establish a state-based marketplace, or cede this authority to CMS to establish a federally facilitated marketplace. Some states had difficulties with the rollout and operation of their marketplaces, and some states that struggled with IT implementation are now using the federal marketplace IT platform. GAO was requested to review CMS's and states' actions to implement the marketplaces. This report (1) describes CMS's actions to assist states that have chosen to transition to a different marketplace IT platform and identify costs and challenges those states incurred in making this transition; (2) assesses CMS's actions taken to assist selected states to ensure that the development and operations of marketplace IT systems can be financially self-sustained; and (3) assesses CMS's steps to monitor the performance of the states' marketplace IT systems. GAO reviewed documentation from CMS and four states selected based on different types of marketplaces, federal grants provided, and enrollment numbers, and interviewed CMS and the states' officials. The Department of Health and Human Services' (HHS) Centers for Medicare & Medicaid Services (CMS) has offered assistance through providing periodic oversight and issuing regulation and guidance to states transitioning from state-based marketplaces to the federally based marketplace IT platform, including two states that GAO reviewed—Hawaii and Oregon—that had made that transition. While CMS provided these states with assistance, documented CMS transition guidance was not finalized until after the two states had completed their transition. The two states incurred costs of approximately $84.3 million, collectively, to transition to the federal platform. The two states' transition efforts included making changes to their Medicaid systems, with these states mainly relying on Medicaid matching funds from CMS to do this. While the selected states successfully transitioned, they encountered challenges during their transitions, due to accelerated transition time frames, difficulties reassigning marketplace responsibilities, delays in receiving approvals from CMS, and trouble accessing historical consumer data in previous vendor-developed marketplace IT systems. CMS took steps to assist Hawaii and Oregon, as well as two states that GAO selected for review that operated state-based marketplaces, Minnesota and New York, in developing plans for marketplace IT system sustainability. CMS assisted these four states by consulting with the states' officials and providing oversight of their sustainability plans, financial audit reports, and risk assessments. However, CMS did not fully ensure the states provided complete sustainability plans and financial audit reports. Further, CMS did not base its risk assessments on fully defined processes. These weaknesses limit CMS's oversight and assurance that it can be informed of the state marketplaces' sustainability efforts. Although CMS established a process to monitor the performance of state-based marketplaces, CMS did not consistently follow its processes. For example, CMS did not ensure that the two selected states, Minnesota and New York, had developed, updated, and followed their performance measurement plans. Also, CMS did not conduct reviews to analyze the operational performance of these states' marketplace IT systems against an established set of parameters. Further, while CMS collected IT performance metrics from the two states, such as the number of electronic enrollments and website traffic volume, it did not link state metrics to goals or establish targets for performance. These weaknesses limit CMS's ability to determine if states' marketplace systems are performing efficiently, effectively, and to provide early warnings of potential problems (see table). GAO recommends that CMS take six actions: ensure that states provide complete sustainability plans; complete financial audit reports; fully define its risk assessment process; complete updated performance measurement plans; align metrics with goals; and conduct operational analysis reviews. HHS concurred with two, partially concurred with two, and did not concur with two of GAO's recommendations, which GAO continues to believe are valid.