Clinical contracts at VA are used to acquire the services of clinical personnel, such as physicians, pharmacists, and nurses. These contracts can be used to fill vacancies for clinicians in specialties that are difficult to recruit, supplement existing VAMC capacity by providing additional clinicians in high-volume areas where VA also manages a staff of its own employees, or fill critical staffing vacancies on a long- or short-term basis. Clinical contracting at VA is governed by two sets of regulations—the FAR and VAAR—that contain general requirements for all VA acquisitions. The FAR contains government-wide regulations and establishes uniform policies and procedures used by all executive branch federal agencies for their acquisitions. VA supplements the FAR with the VAAR to establish uniform policies and procedures for all VA acquisitions of supplies and services. In addition to the VAAR, VA provides guidance to its acquisition workforce in the form of policy directives and handbooks. Both acquisition and clinical staff at VA work together to plan, execute, and monitor clinical contracts at VA. On the acquisition side, COs are responsible for planning, awarding, and administering contracts on behalf of the federal government. Each CO is able to obligate federal funds up to a specified limit and a CO must formally approve all clinical contracts at VA. Common tasks of a CO include developing acquisition planning documents used to begin a clinical contract, conducting market research to determine pricing and availability for a clinical contract, and completing the formal competitive or non-competitive solicitation process for contracts. Each CO works within a network contracting office and is overseen by managers within that office who report directly to VA Central Office. There are 21 network contracting offices throughout VA’s health care system that manage all the contracting activities of a single VISN. Two types of VAMC staff have monitoring responsibilities for clinical contracts—CORs and medical directors. For each VA clinical contract, the CO responsible for the contract designates a COR at the VAMC to help develop the clinical contract and monitor the contract provider’s performance once the provider begins work. Common tasks delegated to the COR include providing input on the performance requirements for the clinical contract, determining how the contract provider’s performance will be measured and monitoring performance once work has begun, validating the contract provider’s invoices to ensure their accuracy, managing contract modifications, and assisting the CO in resolving any issues that may arise with the contract provider. At VA, CORs are commonly administrative personnel responsible for managing the operations of a specialty care line at a VAMC—such as primary care, surgery, etc.—where the contractor will be working. Medical directors in various specialty care lines often assist CORs in monitoring contract provider performance because CORs lack the expertise to evaluate a contract provider’s clinical abilities. Medical directors are responsible for overseeing the clinical care provided by all providers within a specialty care line, including VA-employed and contract clinicians. Medical directors use existing VA processes, including the credentialing and privileging process, to monitor contract providers. This clinical monitoring includes an initial assessment of the clinician’s competency during the first 90 days after the contract provider begins working at a VA facility and typically includes evaluations in several areas—such as patient care, clinical knowledge, and interpersonal communication skills. Upon successful completion of the initial competency evaluation, medical directors monitor contract providers through ongoing evaluations of their performance according to the specific performance standards used to evaluate all clinicians working at VA within their specialty. Medical directors provide the results of these initial and ongoing evaluations to the COR and work with the COR to resolve any issues that may arise as a result of a contract provider’s clinical performance. CORs are responsible for maintaining the official record of the contract provider’s performance and providing official performance assessments to the CO. VA Central Office has primary responsibility for overseeing network contracting offices and manages clinical contracting activities through the VHA Procurement and Logistics Office. There are five primary offices within the VHA Procurement and Logistics Office that are responsible for overseeing various aspects of clinical contracting activities and report to VHA’s Deputy Chief Procurement Officer. (See fig. 1.) Medical Sharing Office. The Medical Sharing Office is responsible for providing guidance to network contracting offices regarding the content and structure of solicitations for clinical contracts and for reviewing several types of clinical contracts. The Medical Sharing Office reviews solicitations of all competitive clinical contracts valued at over $1.5 million, all non-competitive clinical contracts valued at over $500,000, and all organ transplant contracts. All Medical Sharing Office reviews are conducted before a solicitation is issued to ensure that all the necessary provisions are in place prior to any competition or award. Procurement Operations Office. The Procurement Operations Office is responsible for providing ongoing guidance and monitoring of the COR population at VA. The Procurement Operations Office conducts reviews of COR files and publishes a COR newsletter. Procurement Audit Office. The Procurement Audit Office is responsible for ensuring compliance with VA policies and procedures related to contracting. This office conducts internal compliance audits of contracts, including clinical contracts, once they are executed to ensure that all required documentation was included in the final contract and audits the activities of network contracting offices and SAOs to ensure their compliance with VA policies and regulations. Procurement Policy Office. The Procurement Policy Office is responsible for providing guidance to VA’s acquisition workforce in network contracting offices and SAOs. This office produces and updates standard operating procedures for CORs and COs. Service Area Offices. SAOs are the regional contract management entities created to oversee the activities of the 21 network contracting offices and the COs and supervisors that work within them. VHA created three SAOs—East, West, and Central—to manage the contracting activities of six to eight VISNs each. SAOs review solicitations for most clinical contracts during their initial stages to ensure that all necessary provisions are in place prior to any competition or award. We found that all 12 contracts we reviewed from the four VAMCs we visited contained performance requirements consistent with VA acquisition policy. However, we found that 10 of the 12 contracts we reviewed lacked detailed descriptions of contractors’ performance requirements in one or more of the six categories we assessed. We analyzed the content of these contracts using six performance requirement categories we established through reviews of the VAAR, VA policies, and the standards of the leading hospital accreditation organization. (See fig. 2) We also found that the level of detail in contract performance requirements contained in the 12 contracts we reviewed varied both by the type of contract—CBOC contract, specialty care contract, or temporary clinical provider contract—and by the six performance requirement categories we assessed. (See table 1.) The CBOC contracts we reviewed were the most detailed, followed by specialty care contracts and temporary clinical provider contracts. CBOC contracts. Contracts for CBOCs generally contained the most detailed performance requirements. For example, all four of the CBOC contracts we reviewed had complete performance requirements for three of the six applicable categories—the type of provider or care, clinical practice standards, and medical record documentation. However, two of the four CBOC contracts—at the Minneapolis and Seattle VAMCs— lacked detail in one or more of the performance requirement categories we reviewed. For example, the Seattle VAMC CBOC contract was the only CBOC contract we reviewed that did not include a required set of operating hours for the contracted facility, which was a component we assessed within the access to care category. Specialty care contracts. The amount of detail in performance requirements within specialty care contracts varied across our four selected VAMCs. Overall, performance requirements in these contracts lacked specificity in key categories. For example, two of our selected specialty care contracts—for the Seattle and Minneapolis VAMCs—were for the services of cardiothoracic surgeons and had very similar results in our review, except for their descriptions of performance requirements within the clinical practice standards category. For this category, the Seattle VAMC cardiothoracic contract had detailed performance requirements, but the Minneapolis VAMC cardiothoracic contract did not contain a statement describing the provider’s responsibilities for the reporting of and response to adverse events and patient complaints, a component we assessed within the clinical practice standards category.In addition, the Nashville VAMC specialty care contract for the services of a psychiatrist had limited detail because it did not contain requirements for the clinical contractor’s privileges to be renewed at the beginning of each new contract term, and received a partial rating in the medical record documentation category because the contract did not contain performance requirements for the timely entry of information into VA’s electronic medical record. Temporary clinical provider contracts. We found that temporary clinical provider contracts included the least detailed performance requirements among the three contract types we reviewed. For example, the Lebanon VAMC temporary clinical provider contract for a gynecologist was missing descriptions of requirements for providers to undergo VA’s credentialing and privileging process, comply with The Joint Commission requirements that apply to gynecology, and enter relevant data into veterans’ electronic medical records. These specific requirements are components of three separate categories we reviewed—credentialing and privileging, clinical practice standards, and medical record documentation. In addition, the Seattle VAMC temporary clinical provider contract for a pharmacist was missing information on contractor requirements related to both the clinical practice standards and medical record documentation categories. According to VA officials, temporary clinical provider contracts are often written using a format that is designed to allow for shorter and faster contract award timeframes; however, this shorter format likely contributed to less complete performance requirement statements in the contracts we reviewed. We also found the level of detail in contract performance requirements varied significantly by performance requirement category, with a majority of reviewed contracts missing some component of the credentialing and privileging performance requirement category. Credentialing and privileging. This category included a review of three specific components of performance requirements: (1) requirements for all contract providers to initially undergo VA’s credentialing and privileging process; (2) the use of VA’s electronic credentialing system for that In our process; and (3) requirements for the renewal of privileges.review, we found that 10 of the 12 contracts were missing performance requirements from one or more of these three components. Six of the 10 contracts did not contain statements describing the rules for renewal of privileges, which is part of the credentialing and privileging process at VA. Privileges are required to be renewed every 2 years by VA policy. The term of a contract provider’s privileges may not extend beyond the term of the contract regardless of whether the VAMC intends to renew or extend the contract; this means that privileges need to be renewed more often than every 2 years for contract providers depending on the length of the contract. In addition, 5 of the 10 contracts did not contain statements requiring the use of VA’s electronic credentialing system for the credentialing process. Type of provider or care. This performance requirement category had three components: (1) provider licensure requirements, (2) a description of the specialty area care was contracted for, and (3) a listing of key personnel providing services under the contract. contracts we reviewed included detailed performance requirements in this category, with the exception of the Lebanon VAMC temporary clinical provider contract, which did not include the name of the gynecologist selected for the contract or rules for the substitution of another provider if needed. Business processes. This performance requirement category contained three components: (1) invoicing policies; (2) on- and off-duty hours and time card submission; and (3) personnel security responsibilities, such as background checks. Eleven of the 12 contracts included detailed performance requirements in this category, with the exception of the Seattle VAMC CBOC contract, which did not list on- and off-duty hours for staff, but included all other business process components. See Department of Veterans Affairs, Health Care Resources Contracting—Buying, Title 38 U.S.C. 8153, VA Directive 1663 (Aug. 10, 2006). Directive 1663 states that contracts for full-time providers must identify the key personnel proposed to provide the service and their qualifications. standard templates for clinical contracts. VA Central Office officials with whom we spoke said that there is a template for CBOC contracts that has been distributed among contracting officials and that templates for six common specialty care contracts, including cardiology and anesthesiology, are currently in development as of August 2013, and that VA’s progress in developing and implementing these templates is dependent upon VA Central Office and the VA OIG agreeing on the six specialties that will have templates developed. Absent available tools, such as standard templates for common types of contracts—including those for CBOCs, specialty care, and temporary clinical providers—VA cannot reasonably ensure that critical requirements for contract providers’ performance are consistently included in VA clinical contracts and are standardized across VAMCs. Without assurance that these critical performance requirements are included in clinical contracts, VA may not be able to hold clinical contractors accountable for providing the high-quality services VAMCs need to serve veterans. Two of the 12 clinical contracts we reviewed did not include any performance standards and 7 contracts did not include performance standards in key categories. Additionally, the performance standards included within the 12 clinical contracts we reviewed were not always stated clearly and did not always include measurable targets. We found that VA did not provide guidance to COs and CORs on how to develop performance standards that are clear and measurable for determining whether or not clinical contractors met the acceptable quality levels defined in selected contracts. The FAR requires agencies to ensure that requirements for services are clearly defined and appropriate performance standards are developed so that the agency’s requirements can be understood by potential offerors and that performance in accordance with contract terms and conditions In addition, agencies, in order to will meet the agency’s requirements.successfully measure the performance of their operations, use performance standards that demonstrate results, are limited to a vital few activities, and provide useful information for decision-making. To determine whether performance standards used by agencies meet these qualities, we previously found that performance standards should have several attributes—including being clearly stated and using measurable targets. We assessed each of the 12 selected contracts from the four VAMCs we visited to determine if the performance standards for five of the six categories we previously identified included these two attributes of successful performance standards. Performance standards were not clearly stated. Two of the 12 clinical contracts we reviewed—the Lebanon VAMC CBOC and temporary clinical provider contracts—did not include any monitoring plan provisions defining the performance standards against which the VAMC would assess the contractor’s performance. Performance standards included in the monitoring plans of the remaining 10 clinical contracts we reviewed lacked clarity in what would be used to determine a clinical contractor’s performance. For example, the Nashville VAMC CBOC contract includes a performance standard in the clinical practice standards category that requires the contractor to meet a specific target for three of four outpatient satisfaction survey scores. However, VA’s outpatient satisfaction survey includes significantly more than four elements and the contract’s performance standard does not clearly specify which four survey elements will be used for this analysis. In another instance, the Seattle VAMC temporary clinical provider contract includes a performance standard for prescription dispensing that includes multiple standards that cover numerous tasks related to prescription processing and validation in inpatient and outpatient settings. By including multiple parts in a single performance standard, it is unclear what the COR will actually be measuring to determine the clinical contractor’s performance in this area. As table 2 shows, none of the 12 contracts we assessed included clearly stated performance standards in all five categories. It is important to ensure that performance standards included in clinical contracts are clearly stated to minimize confusion during the monitoring of clinical contractors’ performance. Without clearly articulated performance standards, COs and CORs may not be able to effectively assess the performance of clinical contractors and VA may not be able to hold contractors accountable for poor performance. Performance standards did not include measurable targets. We found that there were no categories that included fully measurable performance standards in the 12 contracts we assessed. As previously noted, 2 of the 12 contracts—the Lebanon VAMC CBOC and temporary clinical provider contracts—did not include any monitoring plan provisions defining the performance standards against which the VAMC would assess the contractor’s performance. Of the remaining 10 contracts with monitoring plans, only 1 contract—the Lebanon VAMC specialty contract—included fully measurable performance standards in all categories, and the other 9 contracts either did not include performance standards for a critical category or had performance standards that were only partially measurable in at least one category. (See table 3.) For example, the Nashville VAMC temporary clinical provider contract included a performance standard in the clinical practice standards category that stated that the contractor should have no more than two patient or staff complaints. However, this contract did not define how frequently this target would be measured and therefore it is not possible to determine whether this threshold applies to complaints on a monthly, quarterly, or annual basis. This limited information on how often the complaints would be reviewed and measured for the clinical contractor makes it difficult to determine what a successful performance outcome would be. In another instance, the Minneapolis VAMC specialty contract includes several performance standards in the credentialing and privileging category that indicate the clinical contractor is responsible for maintaining medical and clinical knowledge and engaging in practiced- based learning. However, this contract does not include information on how the clinical contractor’s performance in these areas would be assessed and does not specify a clear numerical target for determining whether or not these standards have been fulfilled. Finally, the Nashville VAMC specialty contract includes a performance standard in the medical record documentation category that states the clinical contractor should maintain documentation in VA’s electronic medical record system of 95 percent, but does not include a description of how the COR would assess compliance with this performance standard. It is important to ensure that the performance standards included in clinical contracts be measurable in order to allow COs and CORs to confidently and effectively measure contractors’ performance. VA has not provided detailed guidance to COs and CORs on how to develop performance standards that allow CORs to conduct meaningful performance monitoring. VA requires that each contract contain appropriate quality assurance standards—including a detailed description of the monitoring procedures that the CO and COR will use to evaluate the performance of clinical contractors and data collection that will be performed. However, this guidance does not include any information on what type of performance standards should be included in a clinical contract, does not include examples of effective performance standards, and does not define the elements of successful performance standards. Without ensuring that COs and CORs have access to detailed guidance on how to construct meaningful performance standards that are both clearly stated and measurable, VA cannot ensure that all clinical contractors will be monitored appropriately or that these performance assessments will be based on the most appropriate measurement of the care they provided to veterans in VA facilities. During our review of 12 clinical contracts at four VAMCs, CORs reported two challenges that may compromise VA’s monitoring of contractors’ performance—the heavy workload associated with the COR position and the lack of adequate training for CORs. CORs at the four VAMCs we visited consistently reported facing significant challenges in effectively carrying out their COR responsibilities for monitoring clinical contractors. Most CORs at the four VAMCs we visited reported that they had other primary duties—including managing staff—that required them to approach their COR responsibilities as a collateral duty. This workload challenge may have led to CORs being unable to effectively monitor clinical contractors. Current VA guidance requires VAMCs to provide CORs with the time to complete their responsibilities and ensure that contract compliance is managed by a knowledgeable COR. Specifically, VA’s standard operating procedure for CORs requires VAMCs to provide CORs with the time and resources necessary to complete required training and fulfill their duties as a COR. Further, the policy governing clinical contracting at VA states that the COR is responsible for ensuring contract compliance and specifies that the COR must have sufficient knowledge of the operation of In addition, the facility and the specific specialty requesting the contract.to monitor clinical contracts effectively CORs are required to perform a number of key functions according to VA’s standard operating procedure for CORs. This guidance requires CORs to submit documentation to the CO and carry out a number of responsibilities as part of their monitoring efforts. (See table 4.) The CORs responsible for monitoring clinical contracts at the VAMCs we visited were often serving in this role as a collateral duty and most had other primary duties that limited their ability to monitor clinical contracts. Thirty-seven of the 40 CORs (93 percent) that completed our data collection instrument, administered to all CORs with responsibility for clinical contracts at the four VAMCs we visited, stated that the COR position is a collateral duty to their primary position. Many of these CORs’ primary positions require them to manage staff, maintain budgets, and oversee other clinical providers. (See table 5.) Based on the results from our data collection instrument administered to 40 CORs at the four VAMCs we visited, we found that the average COR spends about one-quarter of his or her time monitoring approximately 12 contracts, according to estimates provided by CORs; however, some of these CORs were responsible for overseeing significantly more contracts. For example, we found that 6 of these 40 CORs managed nearly 190 of the 452 (41 percent) contracts in place at the four VAMCs we visited and told us they estimated spending at most 30 percent of their work time to their COR duties. These 6 CORs were each managing about 20 more contracts than the average, in our group of 40 CORs completing the data collection instrument, in less time and all 6 of these CORs had other more time-consuming primary duties, such as serving as an administrative officer for a clinical specialty or as a program manager. In addition to CORs who responded to our data collection instrument, during our review of the 12 contracts for various clinical services in the four VAMCs we visited, we found that the CORs responsible for managing these contracts frequently did not have the time to effectively monitor the performance of contract providers. The majority of these CORs said that one of their greatest challenges is not having adequate time to fulfill their COR duties and responsibilities and that the demands of their primary positions prevented them from fully monitoring clinical contractors’ performance. Specifically, CORs for 8 of the 12 contracts reported that the demands of their primary positions have at times prevented them from fully monitoring contract providers’ performance. In addition, CORs for 6 of these 12 contracts stated that they could not complete certain elements of their COR responsibilities due to limited time and resources. For example, one COR stated that she did not adequately monitor costs for the 50 contracts she managed because the duties of her primary position demanded significant portions of her work day and caused frequent interruptions that did not allow her to focus on her duties as a COR. Another COR noted that she only had time to monitor contract issues that were a cause for concern or needed to be addressed immediately. As a result of this time pressure and the demands of their primary positions, CORs reported they did not complete several key monitoring functions, including: (1) submitting required reports, such as quarterly quality assurance reports; (2) documenting contract provider performance issues for annual evaluations; and (3) monitoring contract expenditures. CORs managing 5 of our 12 selected contracts reported that to help address challenges presented by time constraints and the demands of their primary positions, they often worked extended hours in order to complete at least some of their COR duties and manage their day-to-day responsibilities. For example, one COR responsible for managing 1 of our selected contracts, along with 49 other contracts, said that she worked after hours to devote a block of time to her COR work that she could not devote during the normal work day due to the responsibilities of her primary position. In addition, a few CORs also stated that they believe COR duties should be a separate, stand-alone, full-time position rather than a collateral duty to ensure that the monitoring of clinical contractors and other COR duties receives full attention. COs responsible for managing the 12 selected contracts echoed what CORs told us about the heavy workloads and monitoring challenges CORs face. However, none of the COs responsible for the 12 contracts we reviewed directly participate in the monitoring of contract providers’ performance, and all have delegated this responsibility directly to the COR for each clinical contract. Most of the 10 COs responsible for our 12 selected clinical contracts told us that they also rely on the CORs to identify performance issues with contract providers. The majority of these COs recognized that CORs face competing demands and heavy workloads that may prevent them from effectively monitoring contract providers’ performance. Specifically, 7 of the 10 COs responsible for the 12 selected contracts in our review reported that primary position duties and COR duties combined resulted in heavy workloads for CORs. As a result, these COs noted that CORs often do not have enough time to devote to their COR duties. Half of the COs responsible for the 12 selected contracts expressed that the COR responsibility should be a full- time, stand-alone position. We found that VA’s current guidance related to COR responsibilities does not include any information on how VAMCs are to determine the feasibility of whether a COR’s workload—including both COR and primary position responsibilities—will allow them to carry out their tasks as CORs for monitoring contract provider performance. The COR standard operating procedure also does not provide any guidance for determining when COR duties should be assigned as a collateral duty or a full-time responsibility. Without clear guidance on how to determine a COR’s workload, VAMCs can unintentionally assign COR duties to a staff member who does not have the time available to properly monitor clinical contractors. If CORs’ workloads prevent proper monitoring of clinical contracts, VA risks missing the opportunity to proactively identify and correct performance issues with contract providers and to recognize patient safety concerns potentially resulting from contract providers’ actions. By failing to identify performance concerns with contract providers, VA could unknowingly be receiving sub-standard service from these contractors, continue to receive services from these contract providers that do not meet the needs of the VAMCs, and risk patient safety problems when these contracts are extended for additional years. CORs from the four VAMCs we visited noted weaknesses in VA’s COR training courses and our own analysis of these courses confirmed these limitations. Specifically, over half of the 40 CORs that completed our data collection instrument at the four VAMCs we visited responded that either their COR training did not prepare them for their role as a COR or were neutral on whether or not this training was helpful preparation. In addition, CORs for 8 of the 12 contracts we reviewed did not find the required COR training helpful or applicable to VA clinical contracting. Several of the CORs monitoring the 12 selected contracts and CORs managing other contracts at the four VAMCs we visited stated that VA’s required COR training is focused on Department of Defense contracting and is not tailored to CORs managing clinical contracts at VA. For example, one COR stated that the training covered very broad areas of contracts and did not include specific information on which kinds of contracts need detailed quality assurance plans or information on how to manage a clinical contract rather than a supply contract. In addition, a few CORs stated that the instructors for their training courses had limited knowledge of clinical contracting. VA requires CORs to complete training courses to obtain the Federal Acquisition Certification (FAC) for CORs or FAC-COR. are developed and offered by VA’s Acquisition Academy and can be taken both in person and through instructor-led, web-based courses. These courses generally aim to provide individuals with the knowledge and tools to carry out the COR responsibilities, which include the monitoring of clinical contractors’ performance. Previously we found that well-designed training and development programs are linked to both agency goals and to the organizational, occupational, and individual skills and competencies needed for the agency to perform effectively. There are three levels of FAC-COR certifications, which directly correlate with the years of COR’s contracting experience. Specifically, the FAC-COR Level I certification is an 8-hour training and does not require previous experience as a COR, the FAC-COR Level II certification is 40 hours of training (Level I combined with an additional 32 hours of training) and requires 1 year of previous experience serving as a COR, and the FAC-COR Level III certification is 60 hours of training and requires 2 years of previous experience serving as a COR. equipment purchase, and soup contents and production. There were no examples focused on how to evaluate or measure the quality of services provided by a contract provider in a VAMC clinical setting. By focusing on the types of examples in the course content, CORs did not have the opportunity to learn and practice the skills necessary for developing and managing service contracts that do not focus on easily quantifiable elements and involve complex assessments of the performance of clinicians. Included little information on monitoring responsibilities. The course content includes limited information for CORs on post-award monitoring responsibilities for clinical contracts and instead is heavily weighted to discussing the pre-award development of a contract. By failing to incorporate a robust discussion of how CORs should be monitoring contract providers in the course, VA has not provided these critical personnel with a solid foundation for how to conduct their responsibilities for monitoring the performance of contract providers serving in clinical roles at VAMCs. To supplement the required FAC-COR Level II course, VA’s Medical Sharing Office recently developed and implemented an 8-hour training course for CORs managing clinical contracts. However, VA does not currently require this course be completed by all CORs managing clinical contracts. This course covers primarily pre-award contract development responsibilities of CORs and does not include any significant information on the post-award monitoring responsibilities of CORs managing clinical contracts. Without developing required training on clinical contract monitoring, VA cannot ensure that all CORs are receiving adequate information on how to carry out this critical contract management responsibility and cannot ensure that CORs are well trained in how to monitor contract providers. This training places VA at risk of having CORs miss performance weaknesses by clinical contractors that could affect patient safety, and extending clinical contracts for contract providers who do not help VAMCs effectively meet their clinical staffing needs or provide high-quality care to veterans. VA has not established a robust method for overseeing the monitoring of clinical contractors by COs and CORs throughout VA’s health care system. Standards for internal control in the federal government state that agencies should design internal controls that assure ongoing monitoring occurs in the course of normal operations, is continually performed, and is ingrained in agency operations. VA’s primary oversight entity for health care contracting activities, the VHA Procurement and Logistics Office, has a limited role in overseeing the monitoring actions of COs and CORs once a contract has been approved and initiated at a VAMC. The VHA Procurement and Logistics Office conducts limited oversight of contracting activities throughout the VA health care system through its SAOs and Procurement Operations Office. Service Area Offices. According to officials from the three SAOs, the role of the three SAOs in clinical contract monitoring is limited to an audit of the records COs maintain in VA’s electronic Contract Management System. These reviews focus only on the completeness of COs’ electronic contracting files—including documentation that a COR with current training records was assigned to the contract. SAO electronic Contract Management System audits do not include any reviews of CORs’ monitoring of clinical contractors. Procurement Operations Office. The Procurement Operations Office is the only VHA Procurement and Logistics Office entity responsible for overseeing the monitoring activities of CORs; however, the reviews conducted by this office are limited to a remote electronic documentation review of a small sample of COR files. Officials from the Procurement Operations Office told us that to select COR files for these reviews, a Procurement Operations Office staff member aims to select 25 COR files for active contracts per network contracting office—at most 2.1 percent of clinical contracts in an average VISN if all 25 selected COR files are for clinical contracts. VA officials told us that, while the Procurement Operations Office sets a goal to review COR files from two network contracting offices each month, since implementing the program in March 2013 these reviews have been completed in only four network contracting offices and none of these four offices have received feedback on the outcomes of these reviews as of August 2013. Officials added that this limited review schedule is due to only one Procurement Operations Office staff member being assigned to complete these reviews. These reviews also have a narrow focus on the completeness of COR files because the Procurement Operations Office staff member reviewing the files relies on a checklist to verify the presence or absence of required documentation of COR monitoring activities and does not review the quality of information contained within CORs records. The limited review schedule and narrow focus on file completeness do not allow the Procurement Operations Office to comprehensively assess the monitoring activities of COs and CORs throughout VA’s health care system, as recommended by federal internal control standards. Without a robust monitoring system in place, VA cannot reasonably assure that all CORs in all VAMCs are maintaining the proper records of their efforts to monitor the activities of clinical contractors caring for veterans. Clinical contracts serve an important role in helping to ensure that VA can provide health care to our nation’s veterans. Contract providers allow VAMCs and CBOCs to supplement the capacity of VA-employed providers and the ability to provide additional services at these VA facilities. However, VA must maintain robust monitoring of these contract providers to ensure they provide high quality care to veterans and fulfill the responsibilities of their contracts. We identified weaknesses in four areas that limit VA’s ability to effectively monitor clinical contractors and provide support to VAMC-based CORs and contracting officials responsible for conducting these critical reviews. First, the lack of available tools such as templates for common types of clinical contracts requires COs and CORs to rely on less consistent sources for constructing critical performance requirements in clinical contracts. As a result, most clinical contracts we reviewed were insufficiently detailed in their descriptions of contract providers’ performance requirements and these performance requirements were not consistently applied throughout our sample of clinical contracts. Without making tools such as templates available to COs and CORs, VA cannot ensure that all clinical contracts include performance requirements in key categories necessary to VA’s operations and patient care, such as credentialing and privileging and medical record documentation. Second, VA lacks guidance for COs and CORs that defines the types of performance standards and key elements of performance expected to be included in clinical contracts. Without access to this type of guidance, VA cannot ensure that contracting officials have the ability to establish clear and measurable performance standards. Consistency in performance measurement and attention to key elements—including clearly stating performance standards, and ensuring these performance standards can be accurately and consistently measured—is critical to VA’s efforts to ensure that all contract providers are monitored appropriately and can be held accountable for providing quality patient care. Third, two significant challenges CORs face in the monitoring of clinical contracts—COR workload and training—if left unresolved, will likely have continuing negative impacts on the quality of the monitoring that CORs are able to perform. Better guidance to VAMCs on how to determine an appropriate COR workload would allow CORs to fulfill key functions required to effectively monitor clinical contractors’ performance. Currently, VA guidance is silent on how to determine whether or not a potential COR has the time to take on the critical responsibilities of monitoring contract providers. With the current practice of assigning COR duties to VAMC staff as a collateral duty, it is critical that VA define how VAMC staff should allocate COR duties to ensure that contracts are monitored effectively. In addition, CORs are receiving insufficient training to prepare them for monitoring clinical contracts. Currently, CORs for clinical contracts cannot apply the current training to their duties because it focuses on purchasing supplies rather than on acquiring clinical care services. While VA has undertaken efforts to provide supplemental training to CORs on clinical contracts, this additional training fails to address how CORs should monitor clinical contracts and instead focuses on the development of contracts. Without ensuring that COR training addresses all of the critical monitoring functions CORs perform, VA cannot ensure that all CORs are approaching this role with adequate preparation and skills and VA risks failing to identify weaknesses in contract providers’ performance that could affect patient safety. Finally, VA Central Office currently provides limited oversight of clinical contract monitoring activities. Without a robust monitoring program that includes involvement from VA Central Office, VA does not have reasonable assurance that all CORs in VAMCs throughout VA’s health care system are properly monitoring clinical contractors and is missing the opportunity to identify system-wide limitations in the monitoring of these critical contract providers. To improve the monitoring and oversight of clinical contracts, we recommend that the Secretary of Veterans Affairs direct the Under Secretary for Health to take the following five actions: Develop and disseminate tools, such as standard templates, for the most common types of clinical contracts in VA’s health care system. Such tools should include performance requirement statements covering key categories of VA health care policy and guidance—such as credentialing and privileging, provider qualifications, and expectations for compliance with critical VA policies and medical record documentation requirements. Develop and issue guidance on the performance standards that could be included in common types of clinical contracts—including CBOC, specialty, and temporary clinical provider contracts—to ensure that these performance standards are clearly stated in the contracts and have measurable targets for assessing contract provider performance. Revise current standard operating procedures for CORs to provide guidance on the number of contracts, based on size and complexity, each COR should manage to ensure that all CORs maintain a workload that allows them to fulfill their duties as a COR and their primary position responsibilities. Modify existing COR training to ensure it includes examples and discussion of how to develop and monitor service contracts—including contracts for the provision of clinical care in VAMCs. Increase SAO and VA Central Office oversight of COs and CORs by ensuring that post-award contracting files are regularly reviewed for all network contracting offices. VA provided written comments on a draft of this report, which we have reprinted in appendix II. In its comments, VA generally agreed with our conclusions, concurred with our five recommendations, and described the agency’s plans to implement each of our recommendations. In its comments, VA stated that to address our first and second recommendations, VHA’s Medical Sharing Office will work with VHA’s Patient Care Services to update and complete templates for clinical contracts so that they include performance requirement statements covering key categories included in our review. To address our third recommendation, VA noted that VHA’s Procurement Policy and Operations Offices will collaborate to revise existing COR standard operating procedures to include guidance on the number of contracts, based on size and complexity, that each COR should manage to ensure that all CORs maintain a workload that allows them to fulfill their duties as a COR and their primary position responsibilities. To address our fourth recommendation, VA stated that VHA’s Medical Sharing Office will provide training to CORs that covers post-award COR monitoring responsibilities. We support VA’s efforts to provide this training content to CORs, especially those that monitor clinical contracts, but encourage the agency to also modify the content of its existing required COR training to ensure that CORs have the adequate preparation and skills to effectively monitor contract providers. Finally, to address our fifth recommendation, VA noted that VHA’s Procurement Policy and Operations offices will collaborate to revise existing COR standard operating procedures to include Service Area Office and CO oversight to ensure that regular reviews take place for post-award contract files from all network contracting offices. We are sending copies of this report to the Secretary of Veterans Affairs, appropriate congressional committees, and other interested parties. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. If you or your staffs have any questions about this report, please contact me at (202) 512-7114 or at williamsonr@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made major contributions to this report are listed in appendix III. This appendix describes the information and methods we used to examine: (1) the extent to which the Department of Veterans Affairs (VA) establishes complete performance requirements for clinical contractors; (2) the extent to which VA clinical contracts include clear and measurable performance standards for assessing whether or not clinical contractors met the acceptable quality levels defined in selected contracts; (3) challenges VA staff encounter in monitoring clinical contractors’ performance; and (4) the extent to which VA Central Office provides oversight of VA staff responsible for monitoring clinical contracts. Specifically, we discuss our methods for selecting VA medical centers (VAMC) and network contracting offices to review; identifying appropriate VA Central Office Officials to interview; selecting clinical contracts for review and assessment; assessing the performance requirements included in our selected contracts; assessing the performance standards included in our selected contracts; administering a data collection instrument to contracting officer’s representatives (COR) at our selected VAMCs; and evaluating VA’s COR training. We conducted four site visits to VAMCs to obtain the perspectives of VAMC officials responsible for monitoring the day-to-day activities of clinical contractors. To identify VAMCs for our site visits, we: Selected VAMCs that were located in different Veterans Integrated Service Networks (VISN) to ensure that our selected VAMCs varied in their geographic locations and reported to different VISN management officials; Ensured that our selected VAMCs had a variety of active clinical contracts in place to allow for a variety of COR perspectives on clinical contractor monitoring; and Selected at least one VAMC from each of VA’s three Service Area Offices (SAO)—the regional contract management entities created to oversee the activities of the 21 network contracting offices and the contracting officers (CO) and supervisors that work within them. Using these criteria, we selected four VAMCs to visit during our field work located in Lebanon, Pennsylvania; Minneapolis, Minnesota; Nashville, Tennessee; and Seattle, Washington. During our site visits to these locations, we interviewed each VAMC’s leadership team; CORs responsible for managing all active clinical contracts; quality management staff; and the CORs and medical directors responsible for overseeing the clinical contractors for a select sample of clinical contracts. We spoke with these officials about a variety of topics—including COR training and workload, monitoring procedures for clinical contractors, and their role in developing performance requirements and performance standards for clinical contracts. In addition, we spoke with officials from the four network contracting offices responsible for managing the contracting activities of our selected VAMCs to discuss how COs and management staff manage the monitoring of clinical contractors and interact with CORs at VAMCs. Information obtained from our visits to selected VAMCs and interviews with selected network contracting offices cannot be generalized to all VAMCs and network contracting offices throughout VA’s health care system, but provide important insights. We also interviewed VA Central Office officials responsible for developing policies and procedures for VA clinical contracting activities and overseeing the actions of CORs and COs throughout VA’s health care system. We spoke with the following offices within the Veterans Health Administration (VHA) Procurement and Logistics Office: (1) the Medical Sharing Office; (2) the Procurement Audit Office; (3) the Procurement Operations Office; and (4) all three SAOs. To assess the monitoring of clinical contracts at VA, we reviewed the contracts and accompanying COR documentation for a sample of clinical contracts from the four VAMCs we visited during our field work. To select our sample of clinical contracts, we reviewed a list of all active clinical contracts for our selected VAMCs provided by the network contracting offices responsible for overseeing the contracting actions of the selected VAMCs. For each VAMC we visited, we selected three contracts to review in detail—one community-based outpatient clinic (CBOC) contract, one specialty care contract, and one temporary clinical provider contract. (See table 6.) To analyze the performance requirements included in our 12 selected clinical contracts, we identified six performance requirement categories for this analysis by reviewing: (1) VA Acquisition Regulation (VAAR); (2) VA policy, guidance, and training documents; and (3) The Joint Commission’s hospital accreditation standards. We also verified these categories with officials from the Medical Sharing Office, Procurement Operations Office, and Procurement Audit Office to ensure they were an accurate reflection of performance requirements that should be included in VA clinical contracts. During our assessment of the 12 selected contracts, we analyzed the performance requirements included in each clinical contract’s statement of work against these six performance requirement categories and determined whether or not performance requirements existed that matched each categories’ required components. (See table 7.) Results of our reviews were recorded as complete, partial, or incomplete based on whether the contract’s performance requirements covered all, some, or none of the specific components reviewed within each category. To analyze the performance standards included in our 12 selected clinical contracts, we reviewed whether each contract contained monitoring plan provisions that included performance standards in five of the same six categories we identified were vital to VA operations during our review of performance requirements. We did not include the business processes category in our analysis of performance standards because these requirements do not require performance standards for their validation and assessment. For the 12 selected clinical contracts, we grouped performance standards contained in each contract’s monitoring plan under the five performance requirement categories used for this analysis—type of provider or care, credentialing and privileging, clinical practice standards, medical record documentation, and access to care (for CBOC contracts only). We analyzed these performance standards using criteria previously identified by GAO. Specifically, that performance standards should have several attributes—including being clearly stated and using measurable targets. We assessed each of the 12 selected contracts from the four VAMCs we visited to determine if the performance standards included in their monitoring plans met these two attributes of successful performance standards. To determine whether performance standards were clearly stated, two analysts independently determined whether or not each performance standard had specified a desired outcome that was clearly stated and understandable. To determine whether performance standards used measurable targets, two analysts independently determined whether or not each performance standard had a numerical target and included information on the frequency and calculation method for this target. Each analyst entered results as complete, partial, or incomplete for each standard and entered a reason for their determination if the result was partial or incomplete. A third analyst reviewed these assessments and reconciled any differences that arose. We administered a data collection instrument to the 40 CORs responsible for managing a clinical contract at the four VAMCs we visited to determine their perspectives on COR training and workload, and to gather information on whether or not they were serving as a COR as a collateral duty. The data collection instrument included questions on each CORs highest training level, assessment of VA COR training, amount of time devoted to COR duties, and primary position title and description. Each COR completed this data collection instrument in the presence of a GAO analyst during our field work at each of our four selected VAMCs. To evaluate VA’s required COR training, three GAO analysts attended an offering of VA’s COR training in September 2012. These analysts reviewed the content of this training to determine if it met previously identified criteria for successful training—that well-designed training and development programs are linked to both agency goals and to the organizational, occupational, and individual skills and competencies To determine if these needed for the agency to perform effectively.criteria were met, we reviewed the content of this course to determine if it included information that was linked to VA’s agency goals and the organization, occupational, and individual skills needed for a COR monitoring clinical contractors in a VAMC. In addition to the contact named above, Marcia A. Mann, Assistant Director; A. Elizabeth Dobrenz; Pamela Dooley; Carolyn Garvey; Katherine Nicole Laubacher; Laurie Pachter; Said Sariolghalam; Jennifer Whitworth; and Malissa G. Winograd made key contributions to this report.

Summary: VA must frequently contract with non-VA health care providers so that clinical providers are available to meet veterans' health care needs. While recent studies have disclosed problems with VA's development of contracts for clinical services, there has been little scrutiny of how VA monitors and evaluates the care contract providers give to veterans. GAO was asked to review VA's efforts to monitor clinical contractors working in VA facilities. This report examines the extent to which VA establishes complete performance requirements for contract providers, challenges VA staff encounter in monitoring contract providers' performance, and the extent to which VA oversees VAMC staff responsible for monitoring contract providers. GAO reviewed VA acquisition regulations and other guidance. In addition, GAO visited four VAMCs that varied in geographic location and selected a nongeneralizable sample of three types of clinical contracts from each of the four VAMCs to review. GAO discussed how VAMC and VISN staff monitor and oversee these contracts and reviewed contract monitoring documentation. All 12 contracts GAO reviewed from the four Department of Veterans Affairs' (VA) medical centers (VAMC) visited contained performance requirements consistent with VA acquisition policy. However, the performance requirements lacked detail in six categories: type of provider or care; credentialing and privileging; clinical practice standards; medical record documentation; business processes; and access to care. GAO identified these categories from reviews of VA acquisition regulations, VA policies, and hospital accreditation standards; and VA officials verified that these six categories were an accurate reflection of performance requirements that should be in VA clinical contracts. GAO found, for example, one VAMC cardiothoracic contract that had detailed performance requirements while another VAMC's cardiothoracic contract did not contain a statement describing the contract provider's responsibilities for reporting and responding to adverse events and patient complaints. GAO also found that contracting officials lack tools, such as standard templates, that provide examples of the performance requirements that should be included in common types of clinical contracts. Such tools would help ensure consistency in requirements across contracts. Contracting officer's representatives (COR) cited two main challenges in monitoring contract providers' performance--too little time to monitor clinical contractors' performance effectively and inadequate training. Most of the 40 CORs at the four VAMCs in GAO's review said that their clinical contract monitoring duties were a collateral duty and that they had other primary responsibilities, such as serving as a business manager or administrative officer for a specialty clinic within the VAMC. GAO found that, on average, each of these 40 CORs spent about 25 percent of their time monitoring an average of 12 contracts. CORs said the demands of their primary positions at times prevented them from fully monitoring contract providers' performance. Further, VA's current guidance related to COR responsibilities does not include any information on how VAMCs are to determine the feasibility of whether a COR's workload--including both COR and primary position responsibilities--will allow them to carry out their tasks as CORs for monitoring contract provider performance. GAO also found that current VA COR training programs focus on contracts that buy goods, not clinical services, and include little information on monitoring responsibilities. CORs questioned the usefulness of the COR training VA uses to prepare them for monitoring clinical contracts. VA Central Office conducts limited oversight of COR and contract monitoring activities. VA Central Office reviews of COR clinical contractor monitoring activities are limited to a small number of annual file reviews that focus on verifying the presence of required documentation only and do not assess the quality of CORs' monitoring activities. Since implementing the program in March 2013 these reviews have been conducted in 4 of 21 network contracting offices and as of August 2013 none of the 4 offices has received feedback on these reviews. Without a robust monitoring system, VA cannot ensure that all CORs in its VAMCs are properly monitoring, evaluating, and documenting the performance of contract providers caring for veterans. GAO recommends that VA develop and disseminate standard templates that provide examples of performance requirements for clinical contracts, revise guidance for CORs to include workload information, modify COR training, and improve the monitoring and oversight of clinical contracts. VA concurred with GAO's recommendations.