DHS and Coast Guard acquisition guidelines require operational test and evaluation by an independent test agency to confirm that the production configured system meet all requirements before approval for full-rate production. The Coast Guard uses the U.S. Navy’s COTF to conduct operational tests and other evaluations for its major acquisition programs according to those programs’ requirements. COTF serves as an independent evaluator of an asset’s capabilities and has experience testing U.S. Navy assets. In conducting operational testing, COTF evaluates an asset’s operational effectiveness and suitability: For operational effectiveness, testers determine whether or not an asset can meet its missions. For operational suitability, testers determine whether or not the agency can logistically support the asset to an acceptable standard, such as having the asset available for operations 85 percent of its scheduled deployment time. Critical operational issues (COI) are one metric used to determine an asset’s operational effectiveness and suitability and are stated in the form of a question. COIs are examined during testing to evaluate a system’s ability to provide the desired capability and perform its mission. COTF assessed the NSC’s COIs, for example, by comparing the outcome of the test event against the full scope of the COI to determine whether the COI was met or not. Unmet COIs are often the result of related deficiencies, which are identified during testing and include any system that is lacking in its ability to meet normal standards or to function as intended. Deficiencies are scored based on the severity of the problem and its impact on the asset’s ability to accomplish its mission. COIs and deficiencies identified during testing both factor into an asset’s overall operational effectiveness and suitability rating. In addition to verifying that an asset is operationally effective and suitable, operational testing also tests key performance parameters (KPP), which are the capabilities considered essential for mission accomplishment. KPPs are listed by threshold values, which are the minimum acceptable level of performance, and objective values, which are the desired level of performance. For example, a KPP for the NSC is being able to reach a maximum speed of 28 knots for a threshold value and 31 knots for an objective value. KPPs differ from COIs in that KPPs focus on specific performance metrics, while COIs focus on certain types of missions that an asset should be able to conduct or an asset’s ability to be ready to perform those missions. Table 1 provides examples of COIs and KPPs for the NSC. Operational testing can occur over many test events. Two of those key test events are: Initial Operational Test and Evaluation (IOT&E): This event is meant to gather the data necessary to resolve COIs, determine an asset’s operational effectiveness and suitability, and, according to Coast Guard acquisition guidance, occur prior to a full-rate production decision. The test event concludes with a rating of operationally effective or not effective, operationally suitable or not suitable. Follow-on operational test and evaluation (FOT&E): This event is conducted after IOT&E and an asset’s full rate production decision and focuses on refining the conclusions that were made during previous operational test events, evaluating production changes, and reevaluating the system to ensure that it continues to meet operational needs. It also validates any incomplete or deferred requirements and verifies the correction of deficiencies identified during IOT&E. FOT&E concludes with an operational effectiveness and suitability rating similar to that of IOT&E. Following IOT&E and FOT&E, COTF writes a test report that focuses on the resolution of the asset’s COIs and any deficiencies that were identified during testing. These reports typically include a summary of the resolution of the asset’s COIs. As we reported in January 2016, IOT&E took place about 2 years later than planned and after 7 of the 8 planned NSCs were under contract, with 3 operational. We have previously found that delaying critical test events can lead to late discoveries and could result in additional design changes and costs to programs. The 8 NSCs are planned to be fully operational by 2020 and the Coast Guard is phasing out the legacy 378-foot High Endurance Cutters as the NSCs enter operations. During testing, the NSC successfully demonstrated 18 of its 19 COIs, with one COI— cybersecurity—being deferred to FOT&E. This deferral was due to the DHS Director of Operational Test and Evaluation (DOT&E) postponing the testing of the NSC’s cybersecurity capabilities until a more robust test plan could be developed to reflect emerging threats. At the conclusion of IOT&E, COTF found the NSC to be operationally effective and suitable, but with 10 major deficiencies. None of the major deficiencies were rated as severe, which would preclude the NSC from accomplishing its mission. Five of the 10 major deficiencies pertained to the NSC’s weapon systems and cutter boats. Table 2 shows the 10 deficiencies. In its assessment of the NSC’s IOT&E event, DOT&E stated that the reliability and operational availability issues of the weapon systems—the CIWS, NULKA Launcher, TRS-3D air search radar, and the 57-mm gun— affect the overall ability of the NSC to conduct certain missions. While the CIWS, NULKA launcher, and air search radar were all repaired following IOT&E, post-operational reports indicate that problems persist with these systems as they were often unavailable during operations. Despite these findings, as noted above, COTF found the NSC to be operationally effective and suitable. While COIs and deficiencies factor into a system’s operational effectiveness and suitability rating, KPPs are measures of the capabilities considered essential to mission accomplishment. In our January 2016 report, we found that during IOT&E and other test events, the NSC fully met 12 of its 19 KPPs. However, by not meeting all KPPs, the Coast Guard is not able to demonstrate that the NSC is providing the capabilities that it intended to field. For instance, the Coast Guard has not yet demonstrated that the NSC can achieve a hard and soft kill against a subsonic cruise missile as required, or fully meet interoperability requirements with the Department of Defense, DHS, and other government agencies. Table 3 displays the 7 KPPs not fully met for the NSC, the test results, and a discussion of these results. Of the 7 KPPs not met, 3 pertain to the NSC’s cutter boat operations. The cutter boats are designed to be integral to the NSC’s overall capability, operate both within and beyond the visual range of the NSC, and enhance the overall mission effectiveness of the NSC in every mandated mission area. We found that the Coast Guard and COTF have different interpretations of the cutter boat KPPs. COTF conducted IOT&E according to the NSC’s test and evaluation plan and determined that the three KPPs involving cutter boats were not fully met since the boats are unable to operate through all sea state 5 conditions. According to Coast Guard officials, the September 2012 requirements document for the NSC should have been written more clearly to convey the sea state expectation for cutter boat launch and recovery operations, since the NSC’s sea state KPP was never intended to be applicable to the operation of the cutter boats. In January 2016 we recommended, and the Coast Guard concurred, that the NSC’s KPPs for the operation of the cutter boats should be clarified. As we reported in January 2016, according to COTF officials, FOT&E will begin in the fall of 2016 and is scheduled to continue through at least 2017. Following IOT&E, DHS held an acquisition review board (ARB) to discuss the outcome of IOT&E, which resulted in DHS approving the NSC program for full rate production in October 2014. ARBs review major acquisition programs for proper management, oversight, accountability, and alignment with DHS’s strategic functions at acquisition decision events and other meetings as needed. The resulting acquisition decision memorandum (ADM) from October 2014 directed the Coast Guard to conduct FOT&E and complete three action items: (1) complete testing of the cybersecurity COI; (2) verify the correction of all major deficiencies, including the unmet KPPs; and (3) assess the NSC’s cyber-security capabilities. The cybersecurity COI is planned to be tested in 2016, which, if successful, will address the first and third requirements of the ADM, but other testing events are expected to occur through 2017 and possibly beyond. The ADM also directed the Coast Guard to verify the correction of all deficiencies, including the 7 unmet KPPs. According to Coast Guard officials, they have corrected 4 of the 10 major deficiencies from IOT&E that involved equipment failures by restoring the operational status of the related systems on the Stratton, and they have plans to correct four more. However, according to Coast Guard documentation and officials, they may not correct 2 deficiencies due to the cost of making fleet-wide changes, and because the Coast Guard has developed an interim solution. Table 4 shows the Coast Guard’s plans, as we reported in January 2016, for resolving the major deficiencies. As we also found in January 2016, while the Coast Guard has plans to conduct FOT&E for the NSC, it will have accepted the delivery of at least the 6th NSC before the testing is complete, meaning that the Coast Guard will be operating 6 NSCs before it has resolved issues from IOT&E and knows the cutter’s full capabilities. DHS’s guidance for its major acquisitions does not require programs to conduct FOT&E, nor does it specify the timing of FOT&E or the actions that should be taken following the completion of testing.