PPACA directed the federal government to establish and operate a health insurance marketplace, referred to as the federally facilitated marketplace, on behalf of states electing not to establish and operate a marketplace by January 1, 2014. CMS operated a federally facilitated marketplace or partnership marketplace for 34 states for plan years2014 and 2015. Marketplaces, both federal and state, were intended to provide a seamless, single point of access for individuals to enroll in qualified health plans, apply for income-based financial assistance established under the law, and, as applicable, obtain an eligibility determination for other health coverage programs, such as Medicaid or the State Children’s Health Insurance Program (CHIP). PPACA required federal and state marketplaces to be operational on or before January 1, 2014. Healthcare.gov, the public interface for the federally facilitated marketplace, began facilitating enrollments on October 1, 2013, at the beginning of the first annual open enrollment period established by CMS. Since that time, CMS has reported that over 8 million individuals selected a qualified health plan through the federally facilitated marketplace or a state-based marketplace from October 1, 2013, through March 31, 2014. As of October 15, 2014, 6.7 million individuals were enrolled and paying for 2014 health coverage through the marketplaces. HHS estimated up to 9.9 million enrollees for the 2015 enrollment period, which began on According to November 15, 2014, and ended on February 22, 2015.HHS, over 8.4 million people had submitted applications for coverage through the federally facilitated marketplace for the 2015 enrollment period as of January 2, 2015. HHS established the Office of Consumer Information and Insurance Oversight in April 2010 as part of the HHS Office of the Secretary. In January 2011, the office moved to CMS and was renamed the Center for Consumer Information and Insurance Oversight. This office has overall responsibility for providing guidance and oversight for the federal and state systems supporting the establishment and operation of health insurance marketplaces. The Office of Information Services, headed by the CMS Chief Information Officer (CIO), is responsible for oversight of the development and implementation of federal systems supporting the establishment and operation of the federally facilitated marketplace, including review, selection, implementation, and continual evaluation of these systems. The federally facilitated marketplace relies on the Healthcare.gov website and several supporting systems to accomplish enrollment-related activities. To do so, these systems interconnect multiple other systems from a broad range of federal agencies, states, and other entities, such as contractors and issuers of qualified health plans, creating a complex system of systems. The CMS Consumer Information and Insurance Systems Group within the Office of Information Services is tasked with technical oversight of the development and implementation of these systems. A description of each of the major systems for which CMS is responsible for implementing follows. Healthcare.gov is the federal website that serves as the user interface for individuals who wish to obtain coverage through the federal marketplace. Individuals can use the website to obtain information about health coverage, set up a user account, select a health plan, and apply for coverage by the selected health plan. The site supports two major functions: (1) providing information about PPACA health insurance reforms and health insurance options (the “Learn” web page), and (2) facilitating enrollment in coverage (the “Get Insurance” web page). The “Learn” page provides basic information on how the marketplace works, available health plans, and how to apply for coverage. It also contains information on plan costs, ways to reduce out-of-pocket costs, and how individuals can protect themselves from fraud. Individuals do not have to provide personal information to access this section of the website. In contrast to the information-oriented “Learn” page, the “Get Insurance” page allows an individual to take steps to apply for health insurance and other associated benefits. Before an individual can apply for health care coverage or other benefits, CMS must verify his or her identity to help prevent unauthorized disclosure of personal information. The process of verifying an applicant’s identity and establishing a login account is facilitated by CMS’s Enterprise Identity Management system. This system is intended to provide identity and access management services to protect CMS data while ensuring that users’ identities are confirmed, as only authorized users are allowed and capable of accessing CMS resources. The main system, the Federally Facilitated Marketplace (FFM) system, contains several modules that perform key functions related to obtaining health care coverage. The core of the FFM system is a transactional database that was developed to facilitate the eligibility verification process, enrollment process, plan management, financial management services, and other functions, such as quality control and oversight. From a technical perspective, the FFM leverages data processing and storage resources that are available from private sector vendors over the Internet, a type of capability known as cloud-based services. It consists of three major modules: eligibility and enrollment, plan management, and financial management. Eligibility and enrollment module. Individuals seeking to apply for health care coverage through the federally facilitated marketplace use the eligibility and enrollment module to guide them through a step-by- step process to determine their eligibility for coverage and financial assistance. Once eligibility is determined, the applicant is then shown applicable coverage options and has the opportunity to enroll. Throughout the eligibility and enrollment process, the applicant’s information, such as name, address, Social Security number, citizenship status, and employer name, is collected and stored in the FFM system’s database. This information is compared with records maintained by other federal agencies and a private entity to determine whether the applicant is eligible to enroll in a qualified health plan and, if so, to receive the advance payment of the premium tax credit and cost-sharing reductionscost of this coverage. established through PPACA to defray the The module further allows an applicant to view, compare, select, and enroll in a qualified health plan. Options are displayed to the applicant on the Healthcare.gov webpage, and applicants can use the “Plan Compare” function to view and compare plan details. The applicant can customize and filter the plans according to various factors such as plan type, maximum out-of-pocket expenses, deductible, availability of cost-sharing reductions, or insurance company, among others. Once an applicant has signed up for a qualified health plan on Healthcare.gov, information about the enrollment is sent to the chosen health plan issuer. Plan management module. The plan management module is intended to interact with and is primarily used by state agencies and issuers of qualified health plans. The module is intended to provide a suite of services used for submitting, certifying, monitoring, and renewing qualified health plans, as well as managing the withdrawal of these health plans. Specifically, using this module, states and issuers submit “bids” detailing proposed health plans to be offered on Healthcare.gov, including rate and benefits information. CMS then uses the module to review, monitor, and certify or decertify the bids submitted by issuers. Once a bid has been certified and approved for inclusion in the marketplace, it is made available for applicants to enroll through Healthcare.gov. Financial management module. This module is intended to facilitate payments to issuers through electronic transactions. Like plan management, the financial management module is used primarily by issuers of qualified health plans. This module also provides issuers additional services, including payment calculation for reinsurance, risk adjustment analysis, and the data collection required to support these services. Transactions to be supported by the module include payments of premiums and cost-sharing reductions subsidies for individual enrollments, reinsurance, and risk adjustments. The federal Data Services Hub (DSH) acts as a single portal for exchanging information between the FFM and CMS’s external partners, including other federal agencies and state-based marketplaces, for purposes such as facilitating eligibility determinations and transferring plan enrollment information. The DSH was designed as a “private cloud” service supporting various functions such as real-time eligibility queries, transfer of application information, and exchange of enrollment information with issuers of qualified health plans. In conducting Healthcare.gov-related activities, various entities, including federal agencies, a private-sector credit agency, states, issuers of qualified health plans, and agents and brokers connect to and exchange information with the systems supporting the federally facilitated marketplace. Federal agencies such as the Social Security Administration (SSA), Department of Homeland Security (DHS), and Internal Revenue Service (IRS), along with Equifax, Inc. (a private-sector credit agency that CMS contracts with) provide or verify information used in making determinations of a person’s eligibility for coverage and financial assistance. Social Security Administration. This agency’s primary role is to assist CMS in confirming applicant-supplied information by comparing it with information in SSA’s records related to individuals’ citizenship, Social Security number, incarceration status, and death. SSA also provides CMS information on monthly and annual Social Security benefits paid to individuals under the Old Age, Survivors, and Disability Insurance program, if necessary to determine eligibility. Department of Homeland Security. The department assists CMS by verifying the naturalized, acquired, or derivedimmigration status of applicants seeking eligibility to enroll in a qualified health plan or participate in Medicaid, CHIP, or a state-based health plan using information supplied by each applicant through the website. DHS generally undertakes this role only if CMS is unable to verify an applicant’s status with SSA using a Social Security number or if the applicant indicates on the application that he or she is not a U.S. citizen. DHS also assists CMS by verifying the status of noncitizens who are lawfully present in the United States and seeking eligibility to enroll in a qualified health plan or participate in Medicaid, CHIP, or a state-based health plan, as well as current beneficiaries who have had a change in immigration status or whose status may have expired. Internal Revenue Service. IRS provides federal tax information to be used by CMS in determining or assessing income and family size and determining an applicant’s eligibility for insurance affordability programs, including the advance payment of the premium tax credit, cost-sharing reductions, Medicaid, and CHIP. Equifax, Inc. This entity verifies information about an applicant’s current income and employment to assist CMS in making a determination about an applicant’s qualification for insurance affordability programs, such as the advance payment of the premium tax credit and cost-sharing reductions. In addition, several other federal agencies—the Departments of Defense and Veterans Affairs, the Office of Personnel Management, and the Peace Corps—support CMS in determining whether a potential applicant is eligible for or enrolled in minimum essential coverage and therefore may not be eligible to receive the advance payment of the premium tax credit and cost-sharing reductions. For example, applicants that are enrolled in or eligible for coverage under certain government programs such as Medicare or Medicaid, or certain employer-sponsored programs, such as the Federal Employees Health Benefits program, are ineligible for these subsidies. In most states, multiple government systems may need to connect to the FFM system and DSH to carry out a variety of functions related to health care enrollment. For example, most states need to connect their state Medicaid and CHIP agencies to either the FFM system (through the DSH) or their state-based marketplace to exchange data with CMS about enrollment in these programs. In addition, states may need to connect with the IRS (also through the DSH) in order to calculate the maximum amount of advance payments of the premium tax credit. Finally, state- based marketplaces are to send enrollment confirmations to the FFM system so that CMS can administer advance payments of the premium tax credit and cost-sharing reductions and track overall marketplace enrollment. Further, in certain cases, known as partnership marketplaces, states may elect to perform one or both of the plan management and consumer assistance functions while the FFM system performs the rest. The specific functions performed by each partner vary from state to state. Issuers of qualified health plans receive enrollment information from the FFM system using CMS’s Health Insurance Oversight System when an individual completes the application process. In this case, the FFM system transmits the enrollment information to the DSH, which forwards it to the issuer of qualified health plans. The issuer then replies with a confirmation message. Plan issuers also interact with the FFM through the plan management and financial management modules, as previously described. In addition to applicants themselves, agents and brokers may access the Healthcare.gov website to perform enrollment-related activities on behalf of applicants. It is up to individual states to determine whether to allow agents and brokers to carry out these activities, which can include enrolling in health care plans and applying for the advance payment of the premium tax credit and cost-sharing reductions. Figure 1 illustrates the systems that make up the federally facilitated marketplace and their connections with each other, as well as with external partners. In 2014, we reported on challenges CMS and its contractor faced in developing, implementing, and overseeing the Healthcare.gov initiative. We reported on CMS’s efforts to plan and oversee Healthcare.gov- related development contracts, as well as the agency’s efforts in addressing contractor performance, in July 2014. We determined that the agency undertook the development of Healthcare.gov and its related systems without effective planning or oversight practices, despite facing a number of challenges that increased both the level of risk and the need for effective oversight. In addition, CMS incurred significant cost increases, schedule slips, and delayed system functionality for the FFM and DSH systems due primarily to changing requirements that were exacerbated by oversight gaps. Lastly, CMS identified major performance issues with the FFM contractor but took only limited steps to hold the contractor accountable. Specifically, CMS declined to pay about $267,000 in requested fees to the FFM contractor, which was about 2 percent of the $12.5 million in fees paid. We recommended that CMS take actions to assess increasing contract costs and ensure that acquisition strategies are completed and oversight tools are used as required, among other actions. CMS concurred with most of the recommendations. In September 2014 we reported on the planned exchanges of information between the Healthcare.gov website and other organizations, as well as the effectiveness of the programs and controls implemented by CMS to protect the security and privacy of the information and IT systems used to support Healthcare.gov. We described how many systems and entities exchange information to carry out functions that support individuals’ ability to use Healthcare.gov to compare, select, and enroll in private health insurance plans participating in the federal marketplace, as required by the Patient Protection and Affordable Care Act. In addition, we determined that CMS took many steps to protect security and privacy, including developing required security program policies and procedures, establishing interconnection security agreements with its federal and commercial partners, and instituting required privacy protections. However, Healthcare.gov had weaknesses when it was first deployed, including incomplete security plans, lack of a privacy risk analysis, incomplete security tests, and the lack of an alternate processing site to avoid major service disruptions. Further, we identified weaknesses in the technical controls protecting the confidentiality, integrity, and availability of the FFM. Specifically, CMS had not always required or enforced strong password controls, adequately restricted access to the Internet, consistently implemented software patches, and properly configured an administrative network. We made 28 recommendations to HHS to enhance the protection of systems and information related to Healthcare.gov as well as to resolve technical weaknesses in security controls. HHS partially agreed with 3 of the 28 recommendations, agreed with 25, and described plans to implement our technical recommendations. Several problems occurred in the development and deployment of Healthcare.gov and its supporting systems, which affected their performance. These problems included inadequate system capacity, numerous errors in software code, and limited system functionality. Although CMS was aware of these problems prior to initial launch in October 2013, it proceeded with deployment in order to meet this deadline. Consequently, consumers attempting to enroll in health plans were met with confusing error messages, slow load times for forms and pages, and, in some cases, website outages. Since the initial launch of Healthcare.gov and its supporting systems, CMS has taken a number of steps to address these problems, to include increasing system capacity, outlining a new approach for ensuring the quality of software code, and further developing required system functionality. As a result of these efforts, the performance of Heathcare.gov and its supporting systems has improved significantly. Systems supporting Healthcare.gov were initially launched without adequate capacity to accommodate the number of visitors to the website. In particular, when the system was launched on October 1, 2013, the Enterprise Identity Management system was overwhelmed by the number of users attempting to create accounts—nearly half a million in the first 2- and-a-half weeks of open enrollment—preventing the system from functioning as intended. CMS officials within the Office of Information Services stated that they had incorrectly estimated the number of users that would visit the site during the initial launch of the 2014 enrollment period. As a result, CMS had not planned to provide a level of capacity that would ensure uninterrupted service to users in a cost-effective manner. Independent assessments conducted in December 2012 and June 2013 also identified weaknesses in CMS’s capacity planning in the months prior to launch. Examples of these weaknesses included the following: Capacity requirements for hardware for the FFM system were not developed. A plan for capacity for the cloud computing environment had not been developed, and thus there were uncertainties as to whether new and existing system hardware configurations and their performance were adequate to meet existing and proposed system requirements. Existing capacity in the cloud environment was not adequate, and did not include an adequate number of virtual machinesprocessors. Further, in a November 2013 testimony, the CMS Administrator acknowledged that although CMS tried to project demand for the website, the agency underestimated that demand. As a result, consumers attempting to enroll in health plans were met with confusing error messages, slow load times for forms and pages, and in some cases website outages. In particular, due to inadequate system capacity, many consumers experienced difficulty creating accounts, and those that were able to create accounts had difficulty logging into them. Software code for systems supporting Healthcare.gov contained numerous errors, resulting in difficulties in accessing and using the site. For example, in September 2013 (less than 1 month before launch), an IV&V assessment ordered by CMS identified 45 critical and 324 serious code errors across the plan management, financial management, and eligibility and enrollment FFM system modules, with services relating to the eligibility and enrollment module having the highest numbers of errors. Further, the IV&V assessment team reported that there was no evidence that software coding errors were being addressed. Other IV&V assessments of the FFM and DSH systems also noted problems in coding practices used by systems development contractors that indicated concerns about system code. For example, in March 2013, the IV&V assessment team reviewing the FFM and DSH systems noted multiple issues with application coding, including undesirable coding practices that were known to potentially cause errors and the inability of the assessment team to locate CMS or contractor coding standards. CMS also identified concerns with system coding prior to launch. In March 2013, a Director within the Consumer Information and Insurance Systems Group, charged with overseeing the development effort, expressed concerns about the quality of FFM system code during a monthly status meeting. In addition, CMS conducted an assessment of FFM system documentation and development processes in August 2013 and noted that late-stage coding conducted by the FFM system development contractor did not follow expected standards and best practices, resulting in code conflicts between FFM system modules. The assessment further stated that system technical changes and development were being conducted on an ad-hoc basis to resolve production issues rather than being coordinated across development teams. In September 2013, the FFM system development contractor attributed certain coding errors to the urgency of implementing system fixes as quickly as possible. To mitigate these issues, the contractor stated that it was revisiting its code review process to help identify coding errors. However, this action was not timely, as open enrollment began shortly thereafter. Further, in November 2013, the FFM system development contractor, in response to a CMS contracting officer’s concerns about defects and errors in the FFM system code, stated that it was not possible to ensure that each code release addressed all defects because there was not sufficient time to fix the code and retest it to confirm that issues were resolved. CMS officials agreed that some defects were not addressed prior to system launch due to the urgency in meeting the October 1, 2013, deadline. As with the capacity problems, these software code errors also contributed to the problems applicants faced in attempting to enroll in health care plans. For example, according to an HHS report summarizing findings from an Obama administration assessment, for some weeks in the month of October 2013, the Healthcare.gov website was down an estimated 60 percent of the time. In the report, HHS noted that the assessment team determined that hundreds of errors in software code contributed to that downtime. As of initial launch, the functionality provided by the FFM system was limited compared to what was planned, thus hindering users from performing actions needed to compare health plans and small businesses from purchasing plans, as well as requiring the use of a manual process for paying issuers. In September 2011, CMS issued the first FFM system statement of work, which stated that the federal marketplace would provide all exchange capability in states electing not to establish a state-based marketplace. The statement of work identified system modules that were to encompass all federal exchange requirements, including the eligibility and enrollment, plan management, and financial management modules. However, at the time of initial open enrollment in 2013, while parts of the eligibility and enrollment module were completed, others were not. Specifically, after creating an account through the website, consumers could apply for health coverage, compare and select a plan for enrollment, and receive an advance payment of the premium tax credit and Medicaid/CHIP eligibility determination through the eligibility and enrollment module. Nonetheless, consumers were not able to perform other intended eligibility and enrollment functions such as (1) “window shopping” (i.e., comparing different plans) for health plans prior to providing personal information to CMS and signing up for coverage, or (2) designating authorized representatives to apply for coverage on their behalf or change their advance payment of the premium tax credit election. Further, small businesses were unable to purchase health coverage for their employees through the FFM eligibility and enrollment module. Other planned modules, including the plan management and financial management modules, were also not complete and thus did not provide intended functionality. For example, CMS could not use the system to acquire, certify, and manage issuers offering qualified health plans through the exchange’s plan management module. Additionally, the system did not allow payments to be made to health issuers and did not calculate payments for reinsurance through the financial management module. Since the troublesome launch of Healthcare.gov, CMS has taken various actions to address the problems that impeded the initial use of the website and its supporting systems. For example, beginning in October 2013, the agency initiated steps to mitigate the lack of adequate system capacity. Specifically, among other things, it doubled the number of servers for systems supporting Healthcare.gov, added virtual machines for the Enterprise Identity Management and FFM systems, and replaced a virtual database with a high-capacity physical database for the Enterprise Identity Management system, allowing more efficient system processing for both the identity management and FFM systems. By taking these actions, CMS increased overall system capacity to support Internet users—going from 25 to 400 Terabytes of monthly capacity. According to an HHS website, by December 2013, the increased system capacity allowed the system to accommodate more than 1.8 million visits a day from consumers to the website and its supporting systems. According to an HHS progress report issued in December 2013 and other data provided by CMS, Healthcare.gov system availability went from 42.9 percent to just over 93 percent during November 2013, and the FFM system response time went from 8 seconds in late October 2013 to less than 1 second by December 2013. In addition, in October 2013 CMS took steps to mitigate system coding issues. For example, the agency directed its development contractors to, among other things, modify system software to increase the efficiency in system interactions and implement software fixes to address issues with users logging into their accounts. In December 2013, HHS reported that the number of errors encountered by individuals using the system decreased by over 5 percent by the end of November 2013, going from a 6 percent error rate to under 1 percent. Also, CMS documented data quality plans for the Enterprise Identity Management system in March 2014 and the FFM system in June 2014 that outline an approach for improving the quality of the systems’ code. The Enterprise Identity Management system plan calls for peer reviews to ensure that contract requirements are met and product reviews are performed on all deliverables. The FFM plan identifies three types of quality reviews—Peer Reviews, Process and Product Quality Assurance Reviews, and Quality Assessment Reviews—that are to be used to ensure work products conform to documented processes and standards. Peer Reviews. As the primary verification activity, peer reviews are to be conducted to help facilitate early detection of problems, and thus reduce the number of problems discovered in later stages of development, which helps to minimize the cost associated with rework. Peer reviews are to include a review of requirements, design, code, and test planning work products. Peer reviews can be conducted by peer members of the project team or team leads, managers, and design review boards. Process and Product Quality Assurance Reviews. These reviews are intended to ensure that work products, project management processes, high-level development processes, and day-to-day practices adhere to documented CMS processes and standards. These reviews are to be conducted by contractors not directly responsible for the work product or process being reviewed. Quality Assurance Review. The primary purpose of the quality assurance review is to verify that the FFM IT program is progressing based on expectations and is providing business value, and that appropriate risks are identified and managed so that solutions can be delivered on time and within budget. This review is conducted by a contractor Managing Director who is also referred to as a quality assurance Director. These directors are external to the FFM system project, with technical and functional expertise in line with the program. Nonetheless, even with these efforts, IV&V assessments continued to identify issues with software coding practices. For example, in July 2014 the assessment team identified over 11,000 critical code violations in the eligibility and enrollment module of the FFM system which could cause major issues in production or difficulties in maintaining the code. The assessment team highlighted the need for CMS to ensure the FFM system code is reviewed and that critical and major violations are remediated. CMS has also taken steps to develop additional system functionality for the FFM system. In order to complete FFM system development and to improve system functionality already provided by the original contractor tasked with developing this system, the agency awarded a new contract in January 2014. According to the statement of work, this new FFM system development contract represents almost exclusively new development and major fixes to software already developed. The contract called for the new contractor to design, develop, test, and implement services supporting the FFM system. This includes the financial management module, the plan management module, and certain eligibility and enrollment module functions that include eligibility verification and determination. Some FFM system development activities are still in progress, such as the payment service to issuers for subsidy payments to issuers through the financial management module, among others. However, CMS made progress in developing and implementing services related to the FFM eligibility and enrollment and plan management modules. For example, consumers can now “window shop” using the eligibility and enrollment module, and CMS can now use the plan management module to validate plan application information and route the validated information to the appropriate system supporting Healthcare.gov. In developing Healthcare.gov and its supporting systems, CMS did not adhere to best practices for managing IT development projects, which contributed to problems with the launch of Healthcare.gov and its supporting systems. Such best practices include managing requirements to ensure that delivered functionality meets the needs of users, conducting adequate system testing to validate that systems function as intended, and providing oversight to ensure that a project is progressing as planned and that corrective actions are taken as needed. Specifically, CMS did not effectively manage requirements of key systems supporting Healthcare.gov, nor did it adequately test the system, or include key information in system test plans and test cases. In addition, CMS’s oversight of the initiative was limited by an unreliable schedule, lack of estimates of work needed to complete the project, unorganized and outdated project documentation, and inconsistent reviews of project progress. CMS program and contracting officials attributed weaknesses in these IT management areas to the complexity of developing a first-of-its-kind federal marketplace, which was exacerbated by changing requirements and compressed time frames for completing and deploying the systems. CMS has taken action to address deficiencies in applying systems development best practices for the FFM system. However, deficiencies in requirements management, systems testing, and oversight remain. By not engaging in effective systems development practices, CMS lacks essential mechanisms to ensure the successful delivery of IT systems such as Healthcare.gov and its supporting systems. In addition, HHS has not provided adequate oversight of the Healthcare.gov initiative through its office of the CIO, while OMB’s oversight role was limited to facilitating discussions with federal partners, providing federal policy guidance, and overseeing the project’s budget. Best practices developed by the Software Engineering Institute call for, among other things, ensuring that requirements are understood and approved by system stakeholders, including system owners and system developers. Thus, as a project matures and requirements are derived, the requirements should be clearly defined, agreed upon, and approved by the system stakeholders, including system owners and system developers. Consistent with best practices, CMS guidance also requires this approval. Specifically, the CMS Requirements Management Plan documented specifically for the FFM and DSH systems called for functional requirements to be approved by a CMS official—the Center for Consumer Information and Insurance Oversight business owner— before being sent to the development team. The plan further stated that an agency official within the Office of Information Services was to document this approval in the Collaborative Application Lifecycle Tool (CALT), the agency’s project management system and requirements repository. The system records the name of the approver and the date and time at which the requirement was approved. However, in many instances, functional requirements that had been identified for the FFM and DSH systems were included in the development effort prior to or without clear evidence of required CMS approval. Specifically, Of the 37 FFM eligibility and enrollment functional requirements that 9 were designated as having been approved prior to development, 8 were approved after the requirements were sent to 20 were never approved by CMS. Of the 67 DSH functional requirements we selected,approved by a CMS official. CMS officials within the Office of Information Services acknowledged that approvals were not always obtained for functional requirements prior to the development of the FFM and DSH systems. The officials stated that they were unable to enforce consistent application of life-cycle processes because they were trying to develop the system in an expedited fashion to meet the October 2013 deadline. By allowing functional requirements to move to development without approval, CMS did not position itself to ensure that there was a common understanding of requirements between CMS Center for Consumer Information and Insurance Oversight business owners and the contractors tasked with developing these systems, or that expected functionality would be provided. Since Systems Launch, CMS Has Developed a New Requirements Approval Process, but It Is Not Fully Implemented After the initial system launch, CMS documented and began implementing a new IT governance process in June 2014 that calls for business requirementsCMS business owner, the CMS approving authority, and the contract organization’s approving authority—instead of one CMS official (the business owner). In addition, CMS officials within the Office of Information Services stated that functional and technical requirements also require the same three stakeholders’ approval and that these stakeholders’ signatures be included on all requirements documentation, indicating their approval. to be approved by three key stakeholders—the Even with its new requirements approval process, however, CMS has not consistently and appropriately approved requirements. In particular, 1 of 18 FFM system requirements documents that we examined under the new process contained all the necessary approvals for business, functional, and technical requirements that had been documented as part of the effort to improve and expand system functionality. Specifically: Of the 13 business requirements documents, 1 had been fully approved by all three stakeholders. On the other hand, 4 business requirements documents included the signature of the FFM contractor, but did not include the CMS approving authority and business owner signatures; 2 documents were approved by the CMS business owner, but were not approved by the CMS approving authority and the FFM contractor; and the remaining 6 were approved by the CMS approving authority and business owner, but were not approved by the FFM contractor. Of the four functional design documents, none were fully approved by the required stakeholders. Two of the four were not approved by the CMS approving authority and the FFM contractor. One was approved by the CMS business owner and the CMS approving authority, but was missing the approval of the FFM contractor. The remaining functional design document was approved by CMS’s approving authority, but was missing the approval of the FFM contractor and CMS business owner. The one technical design document included the signature of the CMS approving authority, but was missing the signatures of the CMS business owner and FFM contractor. In addition, it was not always clear what requirements were being approved. Specifically, while pages with approval signatures were scanned and uploaded to CALT, 10 of the 18 signature pages were not electronically attached or linked to documents specifying the requirements being approved, making it difficult to determine what requirements were actually approved. These conditions present uncertainty as to whether CMS and its contractors can readily and always determine if the requirements being developed had received the appropriate approval. CMS officials in the Office of Information Services acknowledged the lack of approvals and stated that as of mid-October 2014 they had not yet fully implemented the new IT governance process, which is to include the complete documentation of requirements approvals. Specifically, while CMS has documented the approval procedures for business requirements, it has not yet documented procedures for approving functional and technical requirements. While acknowledging these weaknesses, officials within the Office of Information Services added that CMS is currently tracking approvals through a weekly management report. However, this is inconsistent with the agency’s newly developed procedures, which require stakeholders’ signatures on requirements documentation to indicate approval. The officials further noted that they intend to review all required documentation to identify any signatures that may be missing after 2015 open enrollment is complete. However, this review would take place after the requirements were developed and would not ensure that they were clearly defined, agreed upon, and approved before development began. Until it fully documents and implements its new requirements approval process, CMS may not establish a shared understanding of requirements with its contractors, potentially resulting in critical system functionally not providing needed capabilities. Best practices developed by the Software Engineering Institute call for, among other things, effectively managing requirements by maintaining bidirectional traceability from the high-level original source, such as the business and program requirements, to the lower-level more detailed system and technical requirements, and from those lower-level requirements back to their original source. Such bidirectional traceability allows stakeholders to (1) understand any system-wide effects as a result of changes to requirements, (2) determine whether all high-level requirements have been completely addressed and whether all lower- level more detailed requirements can be traced to a valid source (i.e., maintain requirement dependencies to ensure that higher-level requirements are being addressed by lower-level more detailed requirements), and (3) update requirements documentation as necessary for approved changes. Business processes illustrate the interactions and information exchanges among functional activities and stakeholders (e.g., states, federal agencies, insurers, and employers) performing those activities. These associations provide information for stakeholder relationships and information exchanges to facilitate coordination and agreement among stakeholders concerning their respective roles, responsibilities, and information exchange needs. FFM eligibility and enrollment business process associations were not documented in CALT as required by the Requirements Management Plan. According to CMS officials within the Office of Information Services, these associations were documented in a separate spreadsheet. However, the spreadsheet only included 1,137 of the 3,779 eligibility and enrollment functional requirements. We reviewed all of the 1,137 functional requirements. However, by not maintaining bidirectional traceability among requirements, CMS could not ensure that key stakeholders had a clear understanding of system-wide effects as a result of changes to requirements, determine whether all source requirements had been completely addressed and whether all lower-level requirements could be traced to a valid source, and appropriately update requirements documentation for approved changes. CMS Has Taken Steps to Establish Bidirectional Traceability for Requirements Developed After Initial System Launch To help improve the bidirectional traceability of requirements, CMS documented and began implementing a new FFM requirements management process in June 2014. This process includes guidance on documenting traceability in a new requirements management system— the Quality Center Application Lifecycle Management tool. Since the fall of 2014, CMS and its FFM contractors have made a concerted effort to provide bidirectional traceability within the life-cycle management tool for approved business, functional, and technical requirements for development efforts. In November 2014, FFM contractors, along with CMS officials within the Office of Information Services and Office of Legislation, demonstrated to us how the current process is providing bidirectional traceability. Specifically, contractors provided examples of business requirements and their associated functional requirements using the tool. The contractors also provided examples of how functional requirements and their associated business requirements were linked. According to the FFM contractor, as of November 2014, requirements for three increments within the financial management module and nine increments within the eligibility and enrollment module were fully traceable within the life-cycle management tool. Going forward, effective use of this life-cycle management tool should assist CMS in maintaining bidirectional traceability and, thus, (1) facilitate the understanding of system-wide effects as a result of changes to requirements, (2) help determine whether all source requirements have been completely addressed, and (3) help determine whether all lower- level requirements can be traced to a valid source. Testing an IT system is essential to validate that the system will satisfy the requirements for its intended use and user needs. Effective testing facilitates early detection and correction of software and system anomalies; provides an early assessment of software and system performance; and provides factual information to key stakeholders for determining the business risk of releasing the product in its current state. Best practices developed by the Institute of Electrical and Electronics Engineers (IEEE) suggest that systems testing should be conducted early and often in the life cycle of a systems development project to allow for the modification of products in a timely manner, thereby reducing the overall project and schedule impacts. In May 2011, CMS documented a testing framework that was to establish a consistent, repeatable CMS testing life-cycle process for business application and infrastructure testing. In statements of work, CMS required its FFM and DSH system development contractors to use this framework and perform testing and validation of all software releases prior to implementation. This was to include integration and end-to-end of both the FFM and DSH systems, which would test how, for testing example, various modules of the FFM system work together. This testing would also assess whether the individual systems that support the federally facilitated marketplace work together as intended. Further, CMS testing documentation stated that any critical defects discovered through the testing process were to be corrected or mitigated before the system was put into production. Integration testing is preliminary testing performed by the system developer to assess the interfaces, data, and interoperability of modules and systems within a single business application. End-to-end testing is a type of integration testing that tests all of the business application’s access or touch points, and data, across multiple business applications and systems, front to back (horizontal) and top to bottom (vertical), to ensure business processes are successfully completed. Testing is conducted on a complete, integrated set of business applications and systems to evaluate their compliance with specified requirements, and to evaluate whether the business applications and systems interoperate correctly, pass data and control correctly to one another, and store data correctly. prior to system launch—integration testing with plan issuers that were expected to connect to the DSH to send health plan information to the FFM plan management module had not been completed, with outstanding defects remaining unaddressed for the FFM system eligibility and enrollment module. In addition, end-to-end testing of Healthcare.gov and its supporting systems did not occur prior to system launch as required. Further, CMS did not always ensure that system defects found during the testing were corrected prior to system launch; thus, many defective system components were placed into production. CMS staff within the Office of Information Services, including a Deputy Director, as well as representatives of development contractors for the DSH and FFM systems, stated that there was insufficient time to conduct all the needed testing prior to system launch. This was, in part, because requirements were still being defined in mid-2013 and there were delays in developing software that was ready for testing. Without complete integration and end-to-end testing of the system, CMS lacked a basis for knowing if all Healthcare.gov interconnected systems could operate correctly, pass data correctly to one another, and store data correctly prior to system launch. In addition, without ensuring that defects were corrected prior to placing the system into production, CMS jeopardized its assurance that the system would function as intended. CMS Has Begun Taking Steps to Improve Systems Testing, but Has Not Documented Its New Processes According to officials in the Office of Information Services, CMS has taken steps aimed at improving its testing processes since the highly problematic launch of Healthcare.gov. For example, it has implemented a new tool that integrates systems development and systems testing, which is intended to provide the agency and its contractors greater visibility into the development and testing process. In addition, according to CMS officials in the Office of Information Services, business owners and other stakeholders are now to review key testing documentation to ensure proper test coverage and to validate the results. At the time of our review the agency had not documented this new testing process. Going forward, without a clearly defined and documented process for how CMS will implement the testing tool as well as requirements for stakeholder reviews, CMS may not be able to ensure testing processes are carried out as intended. A key document needed to ensure that testing is carried out effectively is a test plan. Test plans describe the technical and management approach to be followed for testing a system or a component of a system. Best practices, such as those identified by IEEE, call for test plans to identify the test items (software or system) that are the object of testing; provide a description of the overall approach for testing; identify the set of tasks necessary to prepare for and perform testing; identify how testing anomalies will be tracked and resolved; identify roles and responsibilities for individuals or groups responsible for testing; identify the risk issues that may adversely impact successful completion of the planned testing activities; identify the means by which the quality of testing processes will be assured; specify the necessary test environment and test data, such as hardware, software, and test support tools; and specify the criteria to be used to determine whether each test item has passed or failed testing. Test plans we examined for the DSH and FFM systems included most, but not all of the recommended key elements. For example, all 19 DSH and 14 FFM system test plans documented prior to the systems launch in October 2013 identified the test items that were the object of testing; the overall approach for testing; the set of tasks necessary to prepare for and perform the testing; how testing anomalies were to be tracked and resolved; and the roles for individuals or groups responsible for testing. However, a number of these test plans did not address key elements called for by best practices, relating to the quality of testing and the pass/fail testing criteria. Specifically: None of the 19 DSH and 14 FFM system test plans included the means by which quality of testing processes would be assured. Eleven of the 19 DSH and all 14 FFM system test plans were missing detailed criteria to be used to determine whether each test item has passed or failed testing. In addition, these plans varied in the extent to which they addressed risk issues and the test environment information. Specifically: While all 14 FFM system test plans identified risk issues that may adversely impact successful completion of the planned testing activities, 8 of 19 DSH test plans included this information. While all 14 FFM test plans specified the necessary test environment and test data, such as hardware, software, and test support tools, 8 of the 19 DSH test plans included all of the information recommended by best practices. These weaknesses existed, in part, because CMS lacked key elements in its framework. For example, the framework did not require test plans to include the risk issues that may adversely impact successful completion of the planned testing activities; the means by which the quality of testing processes will be assured; or the necessary test environment and test data, such as hardware, software, and test support tools. Further, CMS officials in the Office of Information Services acknowledged the lack of certain key elements in the test plans that existed for systems supporting Healthcare.gov, and attributed this, in part, to an incomplete test plan template. Without including key information in the test plans, CMS had less assurance that testing carried out prior to initial launch was consistently executed and of sufficient quality to validate that systems supporting Healthcare.gov satisfied requirements. Since the initial system launch, CMS has continued to develop test plans for additional FFM system functionality, and these included most, but not all, key elements. Specifically, all 11 post-October 2013 FFM system test plans included test items that are the object of testing; the overall approach for testing; the set of tasks necessary to prepare for and perform testing; how testing anomalies will be tracked and resolved; risk issues that may adversely impact successful completion of the planned testing activities; and, for the most part, specified the necessary test environment and test data, such as hardware, software, and test support tools. Nonetheless, similar to the pre-October 2013 test plans, FFM test plans had not identified all key elements called for by best practices. Specifically, none of the 11 FFM post-October 2013 test plans specified the means by which the quality of testing processes would be assured, and 9 of the 11 test plans lacked criteria to be used to determine whether each test item has passed or failed testing. In addition, these plans varied in the extent to which they discussed roles and responsibilities of individuals or groups responsible for testing. Specifically, while all 11 FFM test plans included the identification of roles for individuals or groups responsible for testing, 5 of these plans did not include the details regarding what tasks these individuals or groups would perform. According to an Information Technology Specialist within the Office of Information Services, the test plan template that was used for test plan development was updated in November 2014 to include the missing key elements we identified. While updating the test plan template with missing elements is a positive step, this will not necessarily ensure key information is included in the test plan. Specifically, although the test plans we reviewed for FFM and DSH included a section for roles and responsibilities, for example, the information included was not always comprehensive and did not provide needed information. As a result, CMS may continue to lack assurance that testing is consistently executed and of sufficient quality to ensure that Healtcare.gov-related systems function as intended. As another key type of testing documentation, test cases describe scenarios that the system must perform to meet intended requirements. Testing teams use these test cases to determine whether an application, system, or a particular system feature is working as intended. Best practices identified by IEEE call for each test case to include a unique identifier so that each test case can be distinguished from all other test cases; specify all outputs and the expected behavior required of the test items; identify dependencies (i.e., other test cases that must be executed before the current test case); identify and describe the objective for the test case (e.g., what feature is being tested); specify the ordered description of the steps to be taken by each participant for the execution of the test procedure; and specify the inputs required to execute each test case (i.e., values, files, databases, etc.). Best practices also state that test cases should be linked to requirements in order to help stakeholders ensure that there is a valid relationship between a system’s requirements and the plans and procedures for testing to ensure they are met. Test cases for components of systems supporting Healthcare.gov included some, but not all key elements. Specifically, all of the selected test cases (42 DSH and 83 FFM) that were documented prior to system launch in October 2013 included a unique identifier. However, these test cases did not always identify two other key elements called for by best practices—outputs and the expected behavior and test case dependencies. Specifically: One of 42 DSH test cases specified outputs and the expected behavior required of the test items. While all 83 of the FFM test cases included expected behavior required of the test items, 12 of the 83 included outputs. One of 42 DSH test cases and 4 of 83 FFM test cases included dependencies. In addition, among the test cases, results were mixed regarding the extent to which they included the objective, the description of steps, and the inputs required. Specifically: While all FFM test cases included the identification and description of the testing objective, 29 of 42 DSH test cases included that information. All the FFM test cases specified the ordered description of the steps to be taken by each participant for the execution of the procedure, but one of the DSH test cases included this information. Among the FFM test cases, 58 of 83 specified all of the inputs required to execute each test case, while none of the DSH test cases did so. In addition, many of the test cases did not include enough information to allow the project team to determine whether the testing contractor had performed the test and whether or not the system passed testing. Specifically, while all 42 DSH test cases included information about whether or not the test passed or failed, 58 of the 83 FFM system test cases were missing pass/fail information. Further, although CMS provided documents that were intended to link requirements to their corresponding test cases, in many instances these documents did not correspond to the test cases we reviewed. Specifically, for 24 of 42 DSH system test cases and 50 of 83 FFM system test cases, the documents did not include enough information to link the requirements being tested and the corresponding test cases. For example, certain documents included a list of test case unique identifiers, but did not include any information about the requirements related to those test cases. In other instances, the documents included test case identifiers that did not use the same naming convention as the test cases we received, so it was unclear as to what test cases those documents were related to. CMS officials in the Office of Information Services acknowledged that test case documentation for systems supporting the initial rollout of Healthcare.gov had been lacking and that there were gaps in the documentation linking the requirements being tested to the corresponding test cases. They attributed these weaknesses to not having always followed required procedures for appropriately documenting test cases. These officials added that the procedures were being followed for the contract awarded in January 2014 for the implementation of additional and enhanced functionality for the FFM system. However, we determined that test cases documented under the new development contract also lacked key elements (as described below). Without key information included in test cases, CMS was limited in its ability to ensure that documented scenarios were performed and thus that applications, systems, or features supporting Healthcare.gov activities were working as intended. Improvements Were Made to Test Cases Developed After Initial System Launch, but Many Still Lacked Key Elements CMS took steps to improve the quality and content of its test cases subsequent to the launch of Healthcare.gov. In particular, all 83 post- October 2013 test cases included a unique identifier, the objective for the test case, the ordered description of steps to be taken by each participant for the execution of the procedure, and expected behavior required of the test items. However, similar to the pre-October 2013 documentation, these test cases did not always include outputs and exact values; test case dependencies; and required inputs. Specifically: 61 of 83 FFM test cases lacked information on outputs and exact 77 of 83 FFM test cases did not include dependencies; and 37 of 83 FFM test cases did not specify all the inputs required to execute each test case. Further, although the newly developed test case documentation did not contain all recommended information, the majority of the documentation did include information to allow the project team to determine whether the testing contractor had executed the test and whether or not the system passed testing, which is a considerable improvement over the previous process. Specifically, the test procedures for 56 of the 70 newly developed test cases that we review were executed and included information about whether the test case passed or failed, compared with 25 of 83 of the pre-launch test cases. In addition, in November 2014 CMS officials in the Office of Information Services and the Office of Legislation, along with representatives from the FFM system development contractor, demonstrated that they were documenting the linkage of requirements to their corresponding test cases within the Quality Center Application Lifecycle Management tool. Going forward, use of this tool should assist CMS in ensuring that there is a valid relationship between test plans, test design, test cases, and test procedures. Nonetheless, until CMS begins to standardize and require all key elements in test case documentation, as recommended by best practices, it may continue lack information needed to determine whether an application, system, or one of its features is working as intended. Best practices that we and the Software Engineering Instituteidentified emphasize the importance of project oversight as a means of ensuring project progress and that appropriate corrective actions can be taken when project performance deviates significantly from the plan. A deviation is significant if, when left unresolved, it precludes the project from meeting its objectives. Best practices call for, among other things, (1) establishing well-constructed schedules that include the entire scope of work activities; (2) estimating the level of effort to be expended by the project team on each task to assist in monitoring the progress of the project; (3) documenting and monitoring activities for managing project documentation; and (4) conducting project progress and milestone have reviews to address performance shortfalls and understand how well requirements are being met. However, CMS did not always (1) ensure project schedules for Healthcare.gov and its supporting systems were well-constructed; (2) estimate level of effort for DSH and FFM functional requirements; (3) implement data management and monitoring processes; and (4) conduct all recommended and required project progress and milestone reviews. CMS officials within the Center for Consumer Information and Insurance Oversight and the Office of Information Services attributed these weaknesses, in part, to challenges with enforcing consistent application of life-cycle processes while trying to develop the system in an expedited fashion to meet the October 2013 deadline. As a result, without adequate and comprehensive information that would be key for understanding the project’s progress, CMS and other oversight agencies may not have the data necessary to appropriately evaluate the project and take corrective actions. A project schedule is a fundamental management tool that specifies when work will be performed in the future and allows for measuring project performance against an approved plan. To this end, our Schedule Assessment Guide states that a project should be guided by an integrated master schedule that reflects the entire scope of work activities. An integrated master schedule may be made up of several or several hundred individual schedules that represent portions of work within a program. These individual schedules are “subprojects” within the larger program. CMS did not always have a comprehensive integrated master schedule prior to system launch in October 2013. For example, IV&V assessment reports issued in December 2012, February 2013, and May 2013 identified weaknesses in project scheduling throughout the Healthcare.gov development process. For example: Activities related to FFM and DSH system implementation and the timeline for the design of the DSH database were not included in the integrated master schedule. Certain key development activities were not included in the FFM integrated project schedule. The FFM testing schedule and the DSH planning schedule did not contain resource assignments needed to complete the work as planned. Therefore, management’s ability to monitor productivity or make effective decisions on the allocation of resources was severely limited. CMS Took Steps to Improve Project Schedules after Initial Launch, but Schedules Were Not Always Well-Constructed After awarding the new FFM development contract in January 2014, CMS re-evaluated project schedules for systems supporting Healthcare.gov. However, project schedules developed since then were not always well- constructed. Best practices identified by usschedules include the following: Logically sequencing all work activities. The schedule should be planned so that critical project dates can be met. To do this, activities need to be logically sequenced—that is, listed in the order in which they are to be carried out. In particular, activities that must be completed before other activities can begin (predecessor work activities), as well as activities that cannot begin until other activities are completed (successor work activities), should be identified. Date constraints and lags should be minimized and justified to help ensure that the interdependence of activities that collectively lead to the completion of events or milestones can be established and used to guide work and measure progress. Confirming that the critical path is valid. The schedule should identify the program critical paththrough the sequence of work activities. Establishing a valid critical path is necessary for examining the effects of any activity’s slipping along this path. The program critical path determines its earliest completion date and focuses the project team’s energy and management’s attention on the activities that will lead to the project’s success. Because a critical path defines a project’s earliest completion date, it must be a continuous sequence of activities from the schedule’s status date to the finish milestone. —the path of longest duration Ensuring reasonable total float. The schedule should identify reasonable total floatdetermined. Large total float on a work activity indicates that the work activity can be delayed without jeopardizing the finish date. The length of delay that can be accommodated without the finish date’s slipping depends on a variety of factors, including the number of date constraints within the schedule and the amount of uncertainty in the duration estimates, but the work activity’s total float provides a reasonable estimate of this value. As a general rule, activities along the critical path have the least total float. so that the schedule’s flexibility can be CMS has made an effort to tie all subprojects into an integrated master schedule and to capture all of the required effort for the Healthcare.gov initiative. Specifically, the agency had documented at least 26 subproject schedules within the integrated master schedule. However, our review of schedules for 4 of 17 FFM subprojects determined that these schedules did not always include key characteristics of a well-constructed schedule. The FFM integrated master schedule contained 17 subproject schedules. We selected 4 schedules that relate to the Plan Management, Small Business Health Options Program, Financial Management, and Eligibility and Enrollment modules of the Federally Facilitated Marketplace System. predecessor activities for 9 percent of its remaining activities. In addition, a significant number of date constraints were reflected in the project schedules, and for the majority of them the agency did not provide a justification. For example, we identified date constraints on 26 percent of the remaining work activities in both the Financial Management and Small Business Health Options Program schedules. CMS did not always ensure that project schedules had a valid critical path. For example, two of the four selected schedules—for the Eligibility Business Operations and the Financial Management projects—did not have valid critical paths because there were several gaps of time where no critical activities were scheduled. Specifically, the critical path for the Eligibility Business Operations schedule had four gaps, ranging from 8 to 15 days, where no critical activities were scheduled. The Financial Management schedule had a gap of nearly 6 months with no critical activities scheduled. In addition, the other two schedules—for the Small Business Health Options Program and Plan Management projects—did not have valid critical paths because the paths were determined by long-duration support and management activities rather than discrete, well-defined work. For example, the Small Business Health Options Program schedule includes management activities such as “Operations Management” and “Deployments Management” that appear in the schedule as critical activities. However, a critical path cannot include these types of activities because, by their very nature, they do not represent discrete effort. CMS did not always ensure reasonable total float. Each of the four project schedules we reviewed appeared to be overly flexible, allowing for many activities to slip a significant number of days before impacting the dates of key events. For example, the Plan Management schedule allowed 50 percent of its remaining activities to slip more than 98 working days before impacting the key finish milestone. Additionally, according to the schedules, remaining activities in the Small Business Health Options Program, Financial Management, and Eligibility Business Operations schedules could be delayed an average of 49 to 50 days before causing the project finish dates to be delayed. Inaccurate values of total float falsely depict true project status, which could lead to decisions that may jeopardize the project. Table 1 below summarizes how well the current subprojects’ schedules met best practices. Because these project schedules did not fully meet key practices for ensuring that they are well-constructed, they are limited as tools for gauging progress and providing reliable estimates of project timelines. In addition, because the reliability of an integrated master schedule depends in part on the reliability of its subordinate schedules, the weaknesses in these schedules will be reflected in the overall schedule for the Healthcare.gov effort. Level-of-effort estimates are used to estimate the amount of time a project will take to develop. According to the Software Engineering Institute, this involves estimating the amount of time and resources to be spent on each work item, such as developing functional requirements for a system. These estimates can then be compared to the actual time and resources expended on each work item. This allows the project’s stakeholders to determine how well the project is progressing and whether schedules should be adjusted or additional resources need to be applied. Consistent with best practices, the CMS Requirements Management Plan documented specifically for the FFM and DSH systems required system development teams to estimate the level of effort for each functional requirement and those estimates to be recorded in CALT. The level-of- effort estimates, according to the plan, were to be used to inform velocity—that is, how quickly the project was being developed. However, CMS and its contractors rarely documented levels of effort for the FFM and DSH functional requirements prior to initial system launch in October 2013. Specifically, nearly 100 percent of the FFM eligibility and enrollment functional requirements and nearly 84 percent of the DSH functional requirements documented prior to initial launch were missing the estimated levels of effort. According to agency officials in the Office of Information Services, contractor earned value management and other financial reports were used in the place of level of effort estimates to track contractor progress. However, the officials agreed that, while these reports would allow them to track the progress made on total project cost estimates, these reports would likely not provide the full insight necessary on how project development was progressing as could be provided with level-of-effort estimates. Due to the lack of level-of-effort estimation, all subsequent monitoring mechanisms that depended on these estimates, including velocity reports, would have provided minimal guidance to CMS and its contractors in monitoring work status and the remaining time needed to complete projects. CMS Has Taken Steps to Estimate Level of Effort for Major System Modules and Supporting Projects, but Has Not Developed or Documented This Policy or Procedures As part of CMS’s efforts to improve project management processes after initial launch of Healthcare.gov and its supporting systems, agency officials stated in August 2014 that they had begun the process of estimating levels of effort and including that information in a system that they historically used to track software defects. They stated that CMS planned to use this system to track further FFM software development efforts, in order to provide more visibility into progress being made by the systems’ development contractors. In addition, the agency provided documentation to demonstrate its progress in estimating level of effort for the FFM system. Specifically, the documentation showed that FFM contractors had begun estimating levels of effort for major system modules and supporting projects. However, current CMS policy does not address estimating level of effort, including how it should be calculated and applied. Specifically, neither CMS’s eXpedited Life Cycle (XLC) process nor its newly developed Requirements Management Guide addresses estimating level of effort at any level. As a result, it will be difficult for agency officials to have reasonable assurance that level-of-effort estimates are developed and calculated and applied in a consistent manner and, therefore, it may be limited as a tool for accurately monitoring progress. state that Best practices identified by the Software Engineering Instituteexplicit specifications should be made concerning what, how, where, and when data should be collected and stored to ensure their validity and to support later use for analysis and documentation purposes. In this case, data are forms of documentation required to support a project in various areas (e.g., administration, configuration management, and quality). These documents, among other things, are then used by project stakeholders to conduct project oversight. Best practices further call for activities for managing these data to be documented and monitored to ensure that data management requirements are being satisfied. Depending on the results of monitoring and changes in project requirements, situation, or status, it may be necessary to re-plan the project’s data management activities. To facilitate a consistent process for managing documents, including those that define requirements, CMS developed a guide in April 2012 for internal and external stakeholders (e.g., other federal agencies providing eligibility determination information). This guide requires the use of CALTSpecifically, the guide calls for updates to the status of each requirement as development progresses to help facilitate project oversight. In addition, the guide provides and defines specific status designations, such as “system requirement approved” and “ready for development.” Further, the agency’s Requirements Management Plan documented specifically for the FFM and DSH systems required that CALT be used for storing various project management documentation, including requirements; source code; network, hardware, and infrastructure descriptions; test cases; test results; and system defects. for managing project data and functional requirements. However, CMS and its contractors did not effectively implement data management processes. For example, they used status designations that were not standardized or defined, which would have hindered CMS’s ability to analyze project progress and effectively oversee the development for the FFM and DSH systems. Specifically: seven undefined status designations, such as “grooming in progress,” were used for the DSH functional requirements; and two undefined status designations, “artifact confirmed” and “planned development completed,” were used for the FFM eligibility and enrollment module functional requirements. Further, key project management documentation was not always stored in CALT as required, which impeded reviews of the development effort. For example, documents needed for reviews by the IV&V assessment team in September 2012 and December 2012, such as quality assurance testing results and hardware and software requirements documents, were located on a contractor’s SharePoint site and were not uploaded to CALT. This would have made it difficult for the assessment team to conduct their review. CMS officials in the Office of Information Services stated that project owners of each individual effort, to include the DSH and the FFM systems, were given autonomy in managing the status of functional requirements within CALT. Consequently, it was difficult for CMS officials responsible for overseeing the entire project to ensure consistency in managing project documentation across each individual project team, of which there were over 200 during the initial development of Healthcare.gov and its supporting systems. The CMS Deputy Chief Information Officer added that because project teams were receiving new requirements well into the development process, required documentation was not always a high priority. This lack of a consistent process for managing project data prior to initial system launch increased the risk that CMS would not have been able to appropriately and effectively (1) monitor the progress of functional requirements as they were being developed, (2) ensure all key documentation needed for overseeing project development activities was documented and updated, and (3) monitor data management. Weaknesses in Data Management Practices Continued after Initial Launch, but CMS Has Plans to Address Them Subsequent to initial system launch, problems in CMS’s data management practices persisted. For example, contractor staff stated that several documents we requested for our review had not yet been uploaded to CALT. Instead, these documents were stored on contractor systems, and thus were not readily available for project oversight. In addition, folders within CALT were not always well-organized, making locating relevant documentation difficult and time consuming. For example, many of the folders were similarly named, or the names of the folders were too vague to determine what documents were included within them. To illustrate, three sub-folders within the same folder were named “UAT.” In addition, while certain software release folders were named by software release number, others were named using a calendar date, making it difficult to know what documentation was relevant to each release. To help mitigate weaknesses in data management monitoring, CMS developed a document management reference guide for the FFM system in July 2014 to establish a process for managing documents created by the FFM development contractor. The guide specified necessary steps for uploading and tracking documents in CALT. In addition, CMS has revised its procedures for tracking the status of requirements through design and testing, and no longer uses undefined status designations. CMS officials in the Office of Information Services stated that, once open enrollment for 2015 has ended, they intend to perform a review of all required CALT documentation, identify missing documents, and locate and upload those documents into CALT. The officials said they expect this effort to be completed by April 2015. According to best practices outlined by the Software Engineering Institute, the purpose of a progress review is to provide relevant stakeholders the results and impacts of a project’s activities and to determine whether there are significant issues or performance shortfalls to be addressed. Milestone reviews are pre-planned events or points in time at which a thorough review of status is conducted to understand how well stakeholder requirements are being met. These reviews are important to ensure that a project is progressing as planned and to identify corrective actions needed. Consistent with best practices, CMS requires progress and milestone reviews for each newly developed system. According to the CMS XLC— its system development life-cycle process—the purpose of these reviews is to provide management and stakeholders with the opportunity to assess project work to date and identify any potential issues. The CMS XLC calls for a project process agreement, which is to serve as an agreement between CMS and its development contractors on the progress and milestone reviews and artifacts (i.e., documentation) required for a project. The agency has identified 11 different progress and milestone reviews which vary depending on the complexity of the project. These reviews are to be conducted by CMS governance boards, which are to approve the project to continue with the next phase of the systems development life cycle. Table 2 describes the progress and milestone reviews documented in the CMS XLC. The FFM, DSH, and Enterprise Identity Management systems were all deemed highly complex by CMS; as such, CMS guidance recommends, but does not require, that they undergo all of the reviews discussed above. However, the three systems did not undergo all the recommended reviews. CMS documented a project process agreement for the Enterprise Identity Management system in January 2012 which stated that it should undergo 10 of the 11 progress and milestone reviews (all but the Investment Selection Review) and specified the required artifacts for each review. However, the agency could not demonstrate that 5 of these reviews were held. CMS officials stated that 4 of these 5 reviews had been performed, but they could not provide any evidence to show this performance. For the DSH and FFM systems, the agency did not document project process agreements, and it provided evidence that some, but not all, of the recommended reviews were held for each. Table 3 shows the recommended reviews for a highly complex system and whether or not those reviews were held for each system. reviews was compromised due to slippages in scheduled deliverables. However, it is unclear whether or not the contractors were aware of the required reviews since the FFM and DSH systems both lacked project process agreements. Regarding the missing review artifacts, the officials further stated that all critical artifacts for each gate review were developed and that the missing artifacts were non-critical. However, the CMS life-cycle framework does not designate artifacts as critical or non-critical, nor does it define these terms. By not ensuring that required progress and milestone reviews took place and that all required artifacts were developed, CMS stakeholders lacked full awareness of the results and impacts of the project’s activities and significant issues or performance shortfalls to be addressed. CMS Has Taken Steps to Improve Processes for Project and Milestone Reviews, but All Required Reviews Have Not Been Held In January 2014, CMS began taking steps to improve its oversight processes for conducting progress and milestone reviews. These improvements, according to officials in the Office of Information Services, included requiring greater collaboration between CMS and its contractors; increasing the number and frequency of contract deliverables, which would include key artifacts provided during the reviews; and placing greater emphasis on progress and milestone reviews as well as formal signoffs prior to the next life-cycle phase. Additionally, in May 2014 and June 2014, CMS documented project process agreements for the portions of the FFM system that were to be developed under the new contract. Despite these efforts, CMS had not documented a project process agreement for DSH as of December 2014. In addition, although Office of Information Services officials stated that they had held all the required reviews for the portions of the FFM system that had been placed into production at the time of our review, they were unable to provide evidence for 5 of 20 required reviews. Table 4 below shows the required reviews for the FFM system and whether or not those reviews were held for newly developed portions of the FFM system that were in production as of July 2014. In addition, CMS was not always following the FFM project process agreement. For example, Office of Information Services officials stated that production readiness reviews and operational readiness reviews were combined for certain increments. However, these reviews have different purposes, and the project process agreements stated that they should occur separately. This approach to conducting reviews puts CMS at continued risk that stakeholders may not be provided sufficient information on the results and impacts of Healthcare.gov-related activities, identify significant issues or performance shortfalls that need to be addressed, and understand how well requirements are being met. In addition, inconsistent application of the project process agreements may lead to key reviews continuing to be missed and approvals not being obtained. We previously reported the lack of certain progress and milestone We reviews in a report on Healthcare.gov contract management.recommended that HHS direct CMS to ensure that information technology projects adhere to requirements for governance board approvals before proceeding with the next phase of the systems development life cycle. HHS agreed with and had begun to take actions to address our recommendation. The Secretary of HHS is required by law and OMB guidance to designate a CIO to be responsible for the management of agency information and information technology. and other assistance to agency heads and other senior management personnel on IT acquisition and management, monitoring the performance of IT programs (including whether to continue, modify, or terminate a program or project), and ensuring compliance with information security requirements. More recently, Congress has reaffirmed the importance of CIOs having a strong role in overseeing IT at executive branch agencies. Specifically, in December 2014, new federal information technology acquisition reform requirements were included in the National Defense Authorization Act, to ensure that the CIO has a significant role in the management, governance, and oversight processes related to their agency’s IT investments. 44 U.S.C. § 3506(a), as amended by Pub. L. No. 104-106, § 5125 (Feb. 10, 1996), and 40 U.S.C. § 11315 (Paperwork Reduction Act of 1995 and the Clinger-Cohen Act of 1996); 44 U.S.C. 3501 note (E-Government Act of 2002, Pub. L. No. 107-347, § 202), and 44 U.S.C. § 3544(a)(3) (Federal Information Security Management Act of 2002), which as of Dec. 18, 2014, was superseded by 44 U.S.C. § 3554(a)(3) (Federal Information Security Modernization Act of 2014, Pub. L. No. 113-283); and OMB, Memorandum for Heads of Executive Departments and Agencies, M-11-29 (Washington, D.C.: Aug. 8, 2011). In March 1996, the Secretary of HHS delegated the Secretary’s IT-related authorities under the Clinger-Cohen Act to the HHS CIO. The CIO in turn requested that operating division heads designate a CIO for their respective divisions, and that the operating division CIOs serve as members of the department’s IT Investment Review Board. This board, which is chaired by the HHS CIO, is to review, validate, and approve selected IT investments in the department’s portfolio. An IT investment may be selected for review at any time during its life cycle if it is high risk and high value, is a high-visibility initiative, or is performing poorly, among other criteria. This is consistent with key practices outlined in our IT investment management guide, which call for the establishment of an enterprise-wide investment review board to be composed of senior executives from IT and business units, who are to be given the responsibility for defining and implementing the organization’s IT investment governance process. Beyond the actions taken by CMS, in August 2011, OMB issued a memorandum to all agency heads, stating that the role of the CIO should be moved away from just policymaking and infrastructure maintenance to true portfolio management for all IT. The memo was intended to clarify the primary responsibility for agency CIOs, to include responsibility over the entire IT portfolio for the agency and for terminating or turning around underperforming investments. Although the Secretary of HHS appointed a CIO, this official had a limited role in overseeing the development and implementation of Healthcare.gov and its supporting systems. The HHS CIO stated that his office did not conduct oversight of the initial design and development for Healthcare.gov and its supporting systems. The CIO further stated that the status of the Healthcare.gov development project was occasionally discussed at regular monthly meetings with senior leadership from each operating division. However, the CIO stated that no issues with Healthcare.gov and its supporting systems were raised in these meetings prior to initial system launch. In addition, although HHS established a process through its IT Investment Review Board that may have revealed technical issues with Healthcare.gov and its supporting systems, the CIO stated that the board has not been active for 2 to 3 years. The CIO also stated that the department is large and federated and his office’s ability to oversee its operating divisions, such as CMS, is limited. He added that oversight reviews are conducted within the operating divisions by their own investment review boards. By not effectively monitoring the performance of the Healthcare.gov initiative prior to the initial launch in October 2013, the HHS CIO was not appropriately positioned to advise the Secretary on actions that should be taken to improve the program. The Office of the CIO Expanded Its Oversight Role after Initial Launch, but a Key Review Board Is Still Not Active The HHS Office of the CIO (OCIO) has expanded its oversight role for the Healthcare.gov initiative since initial launch by convening regular meetings and briefings discussing the Healthcare.gov initiative with officials at various levels. The CIO stated that CMS now regularly shares project documentation with OCIO, which allows them to have better insight as to the status of the project and its development activities. The HHS CIO also stated that although he now has greater insight into the project’s development progress, he does not believe he has the authority to manage IT investments at the operating division level, which includes the Healthcare.gov initiative. However, as previously noted, federal law and OMB guidance place responsibility for overseeing and managing the department’s IT investments with the CIO. Thus, the CIO should be positioned within the department to successfully exercise his authority. Further, the department-wide investment review board called for by HHS policy would provide a mechanism for carrying out these responsibilities, although it has not met for the past 2 to 3 years, according to the CIO. Until the department-wide investment review board carries out its assigned duties, the oversight that HHS provides for Heathcare.gov- related projects may continue to be limited, potentially resulting in missed opportunities to take timely corrective actions on poorly performing projects. By law, OMB oversees the management by federal agencies of information and information technology. OMB’s responsibilities include establishing processes to analyze, track, and evaluate the risks and results of major capital investments in information systems made by executive agencies, as well as issuing guidance on processes for selecting and overseeing agency privacy and security protections for information and information systems. OMB’s guidance under these authorities has included directions to agencies on the roles and responsibilities of CIOs and the establishment of IT investment management processes. In June 2009, OMB launched the Federal IT Dashboard as a public website that reports performance and supporting data for major IT investments. The dashboard is to provide transparency for these investments in order to facilitate public monitoring of government operations and accountability for investment performance by the federal CIOs who oversee them. According to OMB, it began using the dashboard to identify at-risk investments with its launch in June 2009. These investments became the focus of joint OMB-agency TechStat Accountability Sessions (TechStats)—evidence-based reviews intended to increase accountability and transparency and to improve investment performance through concrete actions. In January 2010, OMB began conducting TechStat sessions to enable the federal government to intervene by turning around, halting, or terminating IT projects that are failing or are not producing results. OMB has identified factors that may result in an investment being selected for a TechStat session, such as—but not limited to— evidence of (1) poor performance, (2) unmitigated risks, and (3) misalignment with policies and best practices. Although OMB called for agencies to work with their CIOs to conduct TechStat sessions at the agency level beginning in December 2010, OMB may still select investments for review. Agency CIOs or OMB select these high-risk projects for evaluation, and conduct a review of the proposed improvement plans, revised schedules, and potential changes to budget requests. Although OMB plays a key role in overseeing the implementation and management of federal IT investments, its involvement in overseeing the development efforts of Healthcare.gov and its supporting systems was limited prior to the initial launch in October 2013. According to officials within OMB’s Office of E-Government and Information Technology, headed by the Federal CIO, OMB’s role in overseeing the development of Healthcare.gov and its supporting systems was limited to bringing CMS and its federal partners together to work across technical teams, clarifying federal policy guidance, and overseeing the project’s budget. In particular, OMB facilitated monthly meetings of an IT steering committee consisting of CMS and other key stakeholders (e.g., other federal agencies providing eligibility determination information) that were held to coordinate inter-agency efforts on broader federal marketplace IT work. The meetings, which began in March 2012 and ended in September 2013, primarily focused on addressing key federal marketplace information-sharing policies and identifying barriers to implementation as well as working with federal departments and agencies as necessary on the implementation and execution of the Patient Protection and Affordable Care Act. However, although the Healthcare.gov initiative was considered a high- risk project and independent evaluations and the IT Dashboard identified problems well before its deployment, OMB officials did not select this investment for a TechStat review. Specifically, the dashboard indicated a high-risk evaluation status of Healthcare.gov in March 2013. Officials in the Office of E-Government and Information Technology stated that it was HHS’s responsibility to select the investment for TechStat, but agreed that they retained the right to select investments themselves for review.However, in the case of the Healthcare.gov initiative, OMB did not do so although the IT Dashboard indicated problems 7 months prior to the initial launch of Healthcare.gov and its supporting systems. We reported in 2011 that the Federal IT Dashboard has enhanced OMB’s oversight of federal IT investments. Among other things, we noted that performance data from the dashboard were being used to identify poorly performing investments for executive leadership review sessions. However, in taking steps to oversee the management of the Healthcare.gov IT investment, OMB did not effectively use information provided by this mechanism to analyze, track, and evaluate the risks of this major investment. OMB Took Additional Steps to Provide Oversight by Establishing the U.S. Digital Service Shortly after initial system launch on October 1, 2013, OMB, along with the Federal CIO, assisted HHS and CMS with addressing the technical issues that existed with Healthcare.gov and its supporting systems. Officials in the Office of E-Government and Information Technology stated that after technical issues were reported during initial launch of the system, the role of the Federal CIO was primarily to explore ways to improve the customer experience with the website. In addition, in August 2014, the administration established the U.S. Digital Service, in part to respond to issues with Healthcare.gov and its supporting systems. This service is to collaborate with federal agencies to identify and correct problems with government websites, among other things. OMB’s Deputy Federal CIO serves as the Administrator of the U.S. Digital Service. The mission of this service is to improve and simplify the online experience that people and businesses have with the federal government by establishing standards to bring the government’s digital services in line with the best private sector services; identifying common technology patterns that will help effectively scale services; collaborating with federal agencies to identify and address gaps in their capacity to design, develop, deploy and operate public-facing services; and providing accountability to ensure agencies see results. According to OMB officials in the Office of E-Government and Information Technology, the service is working closely with the CMS systems team charged with developing systems supporting Healthcare.gov. For example, in August 2014, the administration, in conjunction with the U.S. Digital Service, released a set of best practices for effective digital service delivery that are intended to serve as a guide for CMS in further improving systems supporting Healthcare.gov. CMS is working with the service to implement these practices. In addition to its role in assisting CMS with improving the Healthcare.gov initiative through the U.S. Digital Service, OMB’s Office of E-Government and Information Technology continues its role in working with HHS and CMS to oversee the project’s budget. Additionally, the Consolidated and Further Continuing Appropriations Act, 2015 provides for funding to support the Digital Service’s enhanced oversight and guidance for major IT investments. Problems related to insufficient capacity planning, coding errors, and incomplete implementation of planned functionality resulted in numerous performance issues with Healthcare.gov and its supporting systems upon initial launch in October 2013. Consequently, individuals faced significant challenges when attempting to enroll for health insurance coverage. CMS has addressed many of the initial problems by increasing capacity and taking steps to reduce software code errors. Moreover, the agency has been developing additional functionality for the FFM system. Nevertheless, many of the issues arose from the inadequate implementation of key practices for managing IT projects, and these weaknesses had not yet been fully corrected. Specifically, by not managing requirements to ensure that they addressed all needed functionality and not fully documenting and executing key testing activities, CMS did not have reasonable assurance that Healthcare.gov and its supporting systems would perform as intended. In addition, because it did not develop reliable project schedules, measure levels of effort, effectively manage project data, and conduct progress and milestone reviews, CMS had diminished visibility into the project’s status and may have missed opportunities to take corrective actions and avoid problems that occurred upon launch. With the issuance of a new development contract for the FFM system, CMS has taken the opportunity to make improvements in several of these areas. However, until it ensures that it is fully implementing these best practices for managing the development of Healthcare.gov and its supporting systems, it increases the risk that future development will experience additional problems. Further, opportunities exist for HHS to strengthen the involvement of the department’s CIO in conducting oversight of the management of Healthcare.gov and its supporting systems. Until HHS does so it cannot be assured that the implementation and ongoing operation of this high- risk IT investment will continue to provide adequate and sufficient support to millions of Americans seeking to enroll in health care plans through the federally facilitated marketplace. While we previously made recommendations to OMB addressing the use of dashboard ratings for overseeing IT projects’ performance, we found that OMB had a limited role in overseeing the management of the Healthcare.gov IT investment, along with investments in the website’s supporting systems. To improve requirements management for future development covering systems supporting Healthcare.gov, we recommend that the Secretary of Health and Human Services direct the Administrator of the Centers for Medicare & Medicaid Services to direct the Chief Information Officer to take the following two actions: 1. Document the approval process for functional and technical design requirements documentation. 2. Implement the CMS procedure to obtain signatures from the three key stakeholders—the CMS business owner, the CMS approval authority, and the contractor organization approving authority—to ensure that stakeholders have a shared understanding of all business, functional, and technical requirements for systems supporting Healthcare.gov prior to developing them. To improve systems testing processes for future development covering systems supporting Healthcare.gov, we recommend that the Secretary of Health and Human Services direct the Administrator of the Centers for Medicare & Medicaid Services to direct the Chief Information Officer to take the following three actions: 3. Document and approve systems testing policy and procedures, including (1) the use of the system testing tool designed to integrate systems development and systems testing and (2) requirements for stakeholder review of systems test documentation that is intended to ensure proper test coverage and to validate the results. 4. Require key information in system test plans, as recommended by best practices, including the means by which the quality of testing processes will be assured, and the identification of responsibilities for individuals or groups carrying out testing. 5. Require and ensure key information is included in test cases, as recommended by best practices, such as all outputs and exact values; test case dependencies; inputs required to execute each test case; and information about whether each test item has passed or failed testing. To improve oversight processes for systems development activities related to systems supporting Healthcare.gov, we recommend that the Secretary of Health and Human Services direct the Administrator of the Centers for Medicare & Medicaid Services to direct the Chief Information Officer to take the following two actions: 6. Ensure schedules for the Healthcare.gov effort are well constructed by, among other things, (1) logically sequencing activities, (2) confirming the critical paths are valid, and (3) identifying reasonable total float. 7. Develop and implement policy and procedures for estimating level of effort to ensure effort is estimated at the appropriate level (requirements or program area), and define how levels of effort will be used to monitor system development progress. To improve oversight for Healthcare.gov and its supporting systems, we recommend that the Secretary of Health and Human Services direct the HHS Chief Information Officer to carry out authorized oversight responsibilities. Specifically, the Chief Information Officer should ensure the department-wide investment review board is active and carrying out responsibilities for overseeing the performance of high-risk IT investments such as those related to Healthcare.gov. In written comments on a draft of our report (reprinted in appendix II), HHS stated that it concurred with all of the recommendations and identified actions being taken or planned to implement them. Among others, these actions include instituting a process to ensure functional and technical requirements are approved, developing and implementing a unified standard set of approved system testing documents and policies, and providing oversight for Healthcare.gov and its supporting systems through the department-wide investment review board. If the department ensures that these and other actions it identified are effectively implemented, then CMS should be better positioned to more effectively manage current and future systems development efforts for Healthcare.gov and its supporting systems. In addition, the HHS audit liaison provided technical comments from CMS via e-mail. In the comments, CMS disagreed with our characterization of the 11,000 FFM critical code violations that were identified by the IV&V assessment team in July 2014. CMS stated that these code violations were identified very early on in the development phase of building the eligibility and enrollment module and that most of the risk represented by these code violations is to the cost of maintaining the code over time, rather than to its successful functionality. The agency added that any defects which could cause problems with the functionality of the Healthcare.gov system would have been identified and addressed during subsequent testing. However, the IV&V assessment stated that the review was based on a “snapshot of the production code” and not code that was in development. In addition, while the assessment team noted that 328 of the code violations may result in maintainability issues, the team stated that the remaining violations could cause issues in production if not corrected. Other technical comments provided by HHS were incorporated as appropriate. The Chief of Policy, Budget, and Communications within OMB’s Office of E-Government & Information Technology also provided technical comments via e-mail. In the comments, OMB took issue with our statement that it did not conduct a TechStat review when the IT Dashboard indicated problems 7 months prior to the initial launch of Healthcare.gov and its supporting systems. According to the OMB official, a brief dip in the risk rating, such as the one experienced in March 2013, did not necessitate a formal TechStat. The official further stated that the tech surge that OMB instituted shortly after the launch of the system, which included an assessment of its problems, effectively represented a large-scale and comprehensive TechStat session and replaced the need for a separate OMB- or agency-led review. Nevertheless, had such an assessment or a TechStat been conducted earlier in the system development process, the results could have been used to identify and correct deficiencies prior to system launch. We are sending copies of this report to the Secretary of Health and Human Services, the Director of the Office of Management and Budget, and other interested parties. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. Should you or your staffs have questions on matters discussed in this report, please contact me at (202) 512-6304. I can also be reached by e- mail at melvinv@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made major contributions to this report are listed in appendix II. The objectives of this study were to (1) describe the problems encountered in developing and deploying Healthcare.gov and its supporting systems, and determine the status in addressing these deficiencies; and (2) determine the extent to which the Centers for Medicare & Medicaid Services (CMS) oversaw the development effort and applied disciplined systems development practices to manage requirements and conduct systems testing, as well as the extent to which the Department of Health and Human Services (HHS) and the Office of Management and Budget (OMB) provided oversight of the effort. Technical direction letters provide supplementary guidance to contractors regarding tasks contained in their statements of work or change requests. Software Engineering Institute, CMMI for Development, Version 1.3, CMU/SEI-2010-TR- 033 (November 2010, Hanscom AFB, MA). The Software Engineering Institute is a federally funded research and development center operated by Carnegie Mellon University. Its mission is to advance software engineering and related disciplines to ensure the development and operation of systems with predictable and improved cost, schedule, and quality. CALT is CMS’s project management system and requirements repository. The Requirements Management Plan states that requirements should be approved by an official within the Office of Information Services, but that this function can be delegated to other CMS responsible officials. To determine whether requirements maintained bidirectional traceability, we analyzed data extracts of all DSH and eligibility and enrollment module functional requirements from CALT and interdependencies between higher-level and lower-level requirements. We also analyzed requirements documentation developed under the new systems development contract to identify CMS’s current process for maintaining bidirectional traceability. In addition, we interviewed CMS officials as well as DSH system development contractors to obtain an understanding of the requirements management processes, including a live demonstration. With respect to systems testing, we reviewed the CMS testing framework, contract statements of work for the DSH and FFM systems, independent verification and validation reports from September 2012 to July 2014, and system test documentation for these systems. We focused our review on the extent to which CMS applied selected key best practices for software and system (1) test plans and (2) test cases. We assessed all 14 FFM and 19 DSH system test plans documented prior to system launch in October 2013 against best practices identified by the Institute of Electrical and Electronics Engineers that describe key elements that should be included in test plans. In addition, we assessed the 11 FFM system test plans CMS had documented after the new development contract to determine the extent to which these test plans included the key elements identified in best practices. We also assessed DSH and FFM system test cases against best practices identified by the Institute of Electrical and Electronics Engineers that describe key elements that should be included in test cases. In doing so, we analyzed and evaluated all DSH system test cases provided from CMS and documented prior to system launch in October 2013. In addition, we reviewed a non-generalizable random sample of 83 test cases for the FFM system from a population of 585 test cases provided from CMS and documented prior to system launch in October 2013. To determine the extent to which CMS included key elements in test cases developed after the new FFM systems development contract, we reviewed a non-generalizable random sample of 83 test cases from a population of 388. Lastly, we interviewed CMS officials, as well as DSH and FFM system testing contractors, to obtain an understanding of the system testing process. To determine the extent to which CMS, HHS, and OMB oversaw the systems development effort, we obtained and analyzed documentation, such as project schedules, the CMS eXpedited Life Cycle policy, the HHS Enterprise Performance Life Cycle, as well as technical review board presentations and summary letters. We also reviewed project management documentation in CMS’s CALT system. Lastly, we reviewed pertinent oversight laws such as the Clinger-Cohen Act of 1996 and key practices for providing investment oversight that are outlined in GAO’s IT investment management framework. In evaluating the effectiveness of oversight, we focused on (1) project schedules, (2) level-of-effort estimates, (3) data management, and (4) progress and milestone reviews. To determine whether reliable schedules were available to assist with project oversight, we reviewed and analyzed four key subproject schedules for the FFM system, since these subprojects were a major focus of 2014 systems development efforts. Three of the schedules relate to the Plan Management, Small Business Health Options Program, and Financial Management modules of the FFM system, which were planned for initial open enrollment, but had been postponed in August 2013. The fourth schedule related to the eligibility and enrollment module of the FFM system, which is for enrolling individuals for health care coverage. We evaluated the extent to which these schedules were well-constructed as defined in our Schedule Assessment Guide. Our methodology to determine the extent to which project schedules were well-constructed included five levels of compliance. “Fully met” means the program office provided complete evidence that satisfied the elements of the best practice. “Substantially met” means the program office provided evidence that satisfied a large portion of the elements of the best practice. “Partially met” means the program office provided evidence that satisfied about half of the elements of the best practice. “Minimally met” means the program office provided evidence that satisfied a small portion of the elements of the best practice. “Not met” means the program office provided no evidence that satisfied any of the elements of the best practice. To determine the extent to which CMS monitored the project against levels of effort, we reviewed the CMS Requirements Management Plan dated August 2012, and analyzed and evaluated, against the plan, levels of effort documented in the CALT system for all DSH and FFM eligibility and enrollment module functional requirements. For functional requirements developed after the new FFM contract was awarded, we interviewed CMS officials and obtained documentation regarding their efforts in estimating levels of effort for new development. To determine the extent to which CMS monitored data management activities, we reviewed CMS plans and procedures, such as Project Management Plans and the Requirements Management Plan, for managing key project files and functional requirements, and evaluated the extent to which they adhered to CMS plans and procedures within the CALT system. In addition, we reviewed all DSH and FFM eligibility and enrollment module functional requirements contained in CALT to determine the extent to which CMS and its contractor documented key information used for overseeing development progress, such as requirements status fields. To determine whether progress and milestone reviews were conducted in accordance with CMS and HHS policy, we reviewed the eXpedited Life Cycle Process and available project process agreements, and analyzed and evaluated all documentation pertaining to CMS’s progress and milestone reviews for its DSH, FFM, and Enterprise Identity Management systems prior to the October 2013 enrollment. In addition, we reviewed and analyzed progress and milestone reviews held for FFM software releases that were in production as of July 2014 and conducted after the new FFM systems development contract was awarded. Finally, to determine the extent to which CMS, HHS, and OMB provided oversight in the development and implementation of Healthcare.gov and its supporting systems, we interviewed knowledgeable officials, including the CMS Deputy Chief Information Officer, the HHS Chief Information Officer, and officials from OMB’s Office of e-Government and Information Technology. We also obtained documentation and interviewed officials at the Department of Defense, the Department of Homeland Security, the Internal Revenue Service, the Office of Personnel Management, the Peace Corps, the Social Security Administration, and the Department of Veterans Affairs to determine the extent of their role in developing and implementing Healthcare.gov and its supporting systems. To determine the reliability of the data provided from CMS information systems, we performed basic steps to ensure the data provided were valid, and we reviewed relevant information describing these systems. Specifically, we interviewed knowledgeable agency officials within the CMS Office of Information Services about these systems and asked specific questions to understand the controls in place for ensuring the integrity and reliability of the data contained within them. We did not assess the reliability of the systems used to maintain these data or the processes used in extracting the data for our engagement purposed. Based on the results of these efforts, we found the data to be sufficiently reliable for our work. We conducted this performance audit from December 2013 to March 2015 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. In addition to the contact named above, Christie Motley (assistant director), Teresa Tucker (assistant director), James Ashley, Christopher Businsky, Juana Collymore, Nicole Jarvis, Kendrick Johnson, Jason Lee, Jennifer Leotta, Lee McCracken, Thomas Murphy, Constantine Papanastasiou, Andrew Stavisky, and Christy Tyson made key contributions to this report.

Summary: The Patient Protection and Affordable Care Act required the establishment of health insurance marketplaces to assist individuals in obtaining health insurance coverage. CMS, a component of HHS, was responsible for establishing a federally facilitated marketplace for states that elected not to establish their own. This marketplace is supported by an array of IT systems, which are to facilitate enrollment in qualifying health plans. These include Healthcare.gov, the website that serves as the consumer portal to the marketplace, as well as systems for establishing user accounts, verifying eligibility, and facilitating enrollment. GAO was asked to review CMS's management of the development of IT systems supporting the federal marketplace. Its objectives were to (1) describe problems encountered in developing and deploying systems supporting Healthcare.gov and determine the status of efforts to address deficiencies and (2) determine the extent to which CMS applied disciplined practices for managing and overseeing the development effort, and the extent to which HHS and OMB provided oversight. To do this, GAO reviewed program documentation and interviewed relevant CMS and other officials. Several problems with the initial development and deployment of Healthcare.gov and its supporting systems led to consumers encountering widespread performance issues when trying to create accounts and enroll in health plans: Inadequate capacity planning: The Centers for Medicare & Medicaid Services (CMS) did not plan for adequate capacity to support Healthcare.gov and its supporting systems. Software coding errors: CMS and its contractors identified errors in the software code for Healthcare.gov and its supporting systems, but did not adequately correct them prior to launch. Lack of functionality: CMS had not implemented all planned functionality prior to the initial launch of Healthcare.gov and its supporting systems. Since the initial launch, CMS has taken steps to address these problems, including increasing capacity, requiring additional software quality reviews, and awarding a new contract to complete development and improve the functionality of key systems. After it took these actions, performance issues affecting Healthcare.gov and its supporting systems were significantly reduced. In addition, CMS did not consistently apply recognized best practices for system development, which contributed to the problems with the initial launch of Healthcare.gov and its supporting systems. Requirements were not effectively managed: Requirements management helps ensure that a project's plans and work products are aligned with the needs of users. However, CMS did not always ensure that requirements were approved and were linked to source and lower-level requirements. As a result, CMS was hindered in ensuring that expected functionality for the system was delivered. System testing was inconsistent. Testing is essential for ensuring that a system operates as intended. However, Healthcare.gov and its supporting systems were not fully tested prior to launch, and test documentation was missing key elements such as criteria for determining whether a system passed a test. Thus, CMS's assurance that these systems would perform as intended was limited. Project oversight was not effective. Oversight includes monitoring a project's progress and taking corrective actions when its performance deviates from what is planned. However, CMS's oversight was limited by an unreliable schedule, lack of estimates of work needed to complete the project, unorganized and outdated project documentation, and inconsistent reviews of project progress. As it has undertaken further development, CMS has made improvements in some of these areas, by, for example, establishing new requirements management processes and improving test documentation. However, weaknesses remain in its application of requirements, testing, and oversight practices. In addition, the Department of Health and Human Services (HHS) has not provided adequate oversight of the Healthcare.gov initiative through its Office of the Chief Information Officer. The Office of Management and Budget's (OMB) oversight role was limited, and GAO has previously recommended that it improve oversight of IT projects' performance. GAO is recommending that CMS take seven actions to implement improvements in its requirements management, system testing, and project oversight, and that HHS improve its oversight of the Healthcare.gov effort. HHS concurred with all of the recommendations.