Agencies can use a variety of contract types to acquire products and services. Cost-reimbursement contracts are suitable only when uncertainties in the scope of work or cost of services prevent the use of contract types in which prices are fixed, known as fixed-price contracts. A contractor may receive a fixed or base fee on a contract regardless of performance, and also may earn an incentive, which may be used separately or jointly. Such incentive-type contracts, of which award fee contracts are an example, reward contractors with fees based on performance. Award fee contract types are to be used when it is not feasible to devise predetermined objective incentive targets based on cost, technical performance, or schedule, with the focus instead being on subjective criteria, such as project management. In fiscal year 2008, cost- reimbursement contracts made up 94 percent of contracts using award fees. As shown in figure 1, since we issued our report on DOD’s use of award fees, DOD’s use of cost-plus-award-fee (CPAF) contracts has decreased while its use of other cost-type contracts has increased or stayed the same. Figure 2 shows that use of CPAF contracts as a proportion of overall cost- plus contracts varies greatly at the other agencies we reviewed. Contract type is based on a risk assessment by the contractor and the government. The objective is to negotiate a contract type and price (or estimated cost and fee) that will result in reasonable contractor risk and provide the contractor with the greatest incentive for efficient and economical performance. In advance of contract award, outcomes can be identifiable and measurable, identifiable but not measurable, or unable to be identified. The FAR states that an award fee should be used when the work to be performed is such that it is neither feasible nor effective to devise predetermined objective incentive targets applicable to cost, technical performance, or schedule. Alternatively, an incentive fee contract should be used when cost and performance targets are objective and can be predetermined, allowing a formula to adjust the negotiated fee based on variations relative to the targets. These incentive types also can be combined into a multiple-incentive fee contract, which combines objectively and subjectively measured criteria to reward contractor performance while maximizing the government’s ability to use performance metrics that are predetermined, measurable, and targete desired contract outcomes. Agencies, when using multiple-incentive contracts, generally split the available award money into categories that evaluate the contractor’s cost and performance using a combination of objective formulas and subjective judgments to evaluate performance tasks stated in the contract. Appendix II provides definitions contract types as well as terms associated with award fees. Our previous work reviewing the use of award and incentive fees found that programs often paid fees without holding contractors accountable for achieving desired acquisition outcomes, such as meet cost and schedule goals and delivering desired capabilities. Over 5 percent of DOD programs reviewed provided contractors multiple opportunities to earn an estimated $669 million in fees not awarded in previous periods. We also reported that DOD programs regularly paid contractors a significant portion of the available fee for what award fee plans describe as “acceptable, average, expected, good, or satisfactory”performance when federal acquisition regulations and military service guidance state that the purpose of these fees is to motivate excellent performance. To improve the use of award fee contracts we made sev recommendations including suggesting that DOD move toward more outcome-based award fee criteria, ensure that award fees are paid only for above satisfactory performance, and define when rollover is appropria We also recommended that DOD develop a mechanism for capturin award fee d contracts. te. ata for use in evaluating the effectiveness of award fee At NASA, we reported that guidance on the use of CPAF contracts provides criteria for improving the effectiveness of award fees. For example, the guidance emphasizes outcome factors that are good indicators of success in achieving desired results, cautions against using numerous evaluation factors, prohibits rollover of unearned fee, and encourages evaluating the costs and benefits of such contracts before using this contract type. However, we found that NASA did not always follow the preferred approach laid out in its guidance. For example, som evaluation criteria contained input or process factors, such as program planning and organizational management rather than focusing on outcomes or results. Moreover, some contracts included numerous supporting subfactors that may dilute emphasis on any specific criteria. Although the FAR and NASA guidance require considering the costs and e benefits of choosing a CPAF contract, NASA did not perform such analyses. In some cases, we found a significant disconnect between program results and fees paid. In 2007, OMB issued governmentwide guidance highlighting preferred practices and directing agencies to review and update their acquisition policies. That guidance included four fundamental practices: (1) linkingaward fees to acquisition outcomes, (2) limiting the use of rollover, (3) emphasizing excellent performance, and (4) prohibiting payments for unsatisfactory performance. DOD issued new policies on the prope of award fees, while NASA reemphasized its existing guidance. The policies at both agencies reflect these four elements in the OMB guidanc DOE, DHS, and HHS vary in the extent to which they have agencywide guidance, generally allowing operational divisions to supplement award fee guidance. However, existing guidance is not always consistent within agencies or consistent with practices outlined by OMB. e. DOD and NASA ha actions that influenced t d begun to address. Table 1 provides a timeline of he guidance and that have followed its issuance. In March 2006, DOD issued guidance on using award fees that was in direct response to our recommendations. This guidance stated that it is imperative that award fees are linked to desired outcomes such as discrete events or milestones. Such milestones include design reviews and system demonstrations for weapons systems. The guidance also stated that while award fee contracts should be structured to motivate excellent contractor performance, award fees must be commensurate with contractor performance over a range from satisfactory to excellent performance. The guidance recognized that performance that is less than satisfactory is not entitled to any award fee and that satisfactory performance should earn considerably less than excellent performance, otherwise the motiva achieve excellence is negated. Further, the guidance established tha practice of rolling over unearned award fees from one period to another should be limited to exceptional circumstances. The guidance also established the Award and Incentive Fee Co mmunity of Practice to facilitate discussion of strategies across the acquisition workforce and serve as a repository for policy information, related training courses, and examples of good award fee arrangements. In October 2006, Congress required DOD to develop specific guidance linking award and incentive fees to acquisition outcomes. The requirement specified that among other elements, the guidance should define acquisition outcomes in terms of program cost, schedule, and performance and provide guidance on determining ‘‘excellent’’ or ‘‘superior’’ performance. Additionally, the guidance was to prohibit the payment of award fees for performance that is judged to be below satisfactory or does not meet the basic requirements of the contract. The guidance was also to establish standards for determining the percentage of the available award fee, if any, for various levels of performance ranging from satisfactory to excellent. Further, DOD was to provide specific guidance on the circumstances, if any, in which it may be appropriate to roll over award fees that are not earned in one award fee period to a subsequent award fee period or periods and include performance measures to evaluate the effectiveness of award and incentive fees as a tool for improving contractor performance and achieving desired program outcomes. Finally, DOD’s guidance was to provide mechanisms for sharing proven incentive strategies for the acquisition of different types of products and services. on In April 2007, DOD responded by providing additional guidance that reemphasized that cost-plus award fee contracts are suitable for use when it is neither feasible nor effective to devise objective targets applicable to cost, technical performance, or schedule. Recognizing that most DOD contracts contain objective criteria, the guidance clarified that in insta where objective criteria exist and the Contracting Officer and Program Manager wish to also evaluate and incentivize subjective elements of performance, the most appropriate contract type would be a multiple incentive type contract containing both incentive and award fee criteria. Additionally, the guidance defined the levels of performance used to evaluate contractors and the corresponding percentage of fee that could be earned. Table 2 illustrates the scale as recommended by DOD. To address the use of award fees and specific weaknesses previously identified by its Inspector General in the early 1990s, NASA issued guidance in its FAR Supplement and Award Fee Contracting Guide. Previously identified weaknesses included the awarding of excessive fees with limited emphasis on acquisition outcomes (end results, product performance, and cost control), rollover of unearned fee, use of base fee, and the failure to use both positive and negative incentives. NASA updated its award fee guide in 1994, 1997, and 2001 to explain and elaborate on award fee policy. The 2001 revision also reflects the FAR’s additional emphasis on using performance-based contracts. NASA’s Award Fee Contracting Guide provides a tool to contracting officers with guidance on when to use an award fee contract, the risk involved with various contract types, and how to combine award fees with other contract types. Additionally, NASA’s guidance addresses award fee practices that are designed to produce positive results. For example, in describing evaluation factors to be used in award fee determinations, NASA established a preference to use outcome factors when feasible since they are better indicators of success relative to the desired result. Additionally, the guidance provides the scale displayed in table 3 with which to evaluate contractor performance and emphasizes that no award fee will be paid to contractors that perform unsatisfactorily. (percentage) Of exceptional merit; exemplary performance in a timely, efficient and economical manner; very minor (if any) deficiencies with no adverse effect on overall performance. Very effective performance, fully responsive to contract requirements; contract requirements accomplished in a timely, efficient and economical manner for the most part; only minor deficiencies. Effective performance; fully responsive to contract requirements; reportable deficiencies, but with little identifiable effect on overall performance. Meets or slightly exceeds mi deficiencies with identifiab nimum acceptable standards; adequate results; reportable le, but not substantial, effects on overall performance. Does not meet minimum acceptable standards in one or more areas; remedial action required in one or more areas; deficiencies in one or more areas which adversely affect overall performance. DOE, HHS, and DHS varied in the extent to which they had existing guidance specific to award fees and the extent to which that guidance was consistent acquisition officers and senior procurement executives in December 2007, with OMB guidance. While OMB’s guidance was sent to chief many officials with whom we met across various levels at several agencies its within these departments were unaware of the OMB guidance memo or contents. DOE has supplemental guidance to the FAR that outlines how award fee should be considered in contracts for operations and management and separately for lab contracts. Recognizing the complexity of this guid DOE created implementing guidance specific to management and operations contracts in September 2008 that links performance fees to acquisition outcomes and limits the use of rollover. Specifically, the guidance states that fee must relate to clearly defined performance objectives and performance measures. Where feasible, the performance objectives and measures should be expressed as desired results or outcomes. It also states that following these principles will increase the probability that the contractor will only receive performance fee for government-negotiated acquisition outcomes. Additionally, the departmental guidance states that rollover should be used in limited circumstances where convincing evidence of the cost and benefit are considered by a senior procurement executive. The guidance acknowledges that allowing the contractor a second chance at earning th same fee could undermine the incentive in the original award fee period. In response to this concern, the guidance states that if rollover is used, t he contractor can only earn a portion of the unearned fee based on how the contractor came to delivering the originally negotiated performance (for example, a contractor failing to reach a milestone by a year must earn significantly less than a contractor that fails by a week) and how much DOE still desires the originally negotiated performance, some other performance, or both. While linking fee to acquisition outcomes and limiting the use of rollover are in line with OMB’s guidance, several other elements of DOE’s departmental guidance are not. For example, both DOE’s supplemental acquisition policy and the implementing guidance establish CPAF contracts as generally the appropriate type of contract for management and operations. The OMB guidance states that in using an award fee contract, contracting officers should conduct and document risk benefit analyses that support use of the contract type. As part of this analysis, they are to conduct a risk assessment and ensure that incentive strategies are consistent with the level of risk assumed by the contractor and motivate the contractor by balancing awards with negative consequences. Also, according to both the OMB memo and the FAR, contracting officers should determine whether administrative costs associated with managing the incentive fee are outweighed by the expected benefits. Further, agencies should ensure sufficient staff are available to properly structure and monitor the contract. These factors require a case by case consideration before using an award fee contract which contradicts DOE guidance that suggests the general application certain type of contract for work of a particular type. Additionally, the DOE departmental guidance does not clearly def standards of performance for each rating category (e.g., satisfactory, above satisfactory, excellent) or the percentage of fee the contractor should be paid for each of these rating categories as stated in OMB’s E guidance, as do DOD and NASA. Instead, some divisions of DO own (including the Office of Science and NNSA) have developed their standards and methods of evaluation. These standards varied among contracts at the sites. For example, at a multimission site, some contracts prohibited payment of fee to contractors that did not perform ine the satisfactorily while others allowed a reduced fee for that level of performance. DOE contracting officials at the division level told us that while they appreciate the flexibility allowed in coming up with their own evaluation criteria, they could benefit from additional departmental guidance on performing the evaluations and establishing standards. DHS provides guidance on award fees in its acquisition manual, but does not fully address the issues in the OMB guidance. The DHS guidance requires award fee plans to include criteria related (at a minimum) to schedule, and performance. Further, it establishes that award fees are to be earned for successful outcomes and no award fee may be earned against cost, schedule or performance criteria that are ranked below “successful” or “satisfactory” during an award-fee evaluation of contracto performance. However, the manual does not describe standards or definitions for determining various levels of performance. Additionally, it does not include any limitation on the use of rollover. DHS procurement officials noted that there is a need for better guidance on the use of award fees. They also noted, however, that the extent of that need will largely be determined by the pending interim FAR rule on award fees. In response to revised guidance, some DOD components reduced costs and improved management of award fee contracts by limiting the use of rollovers and by tying fees more directly to acquisition outcomes. Potential changes at NASA —such as documented cost-benefit analyses and adding the management of award fee contracts as an area of review— are too recent for their full effects to be judged. At DOE, DHS, and HHS, individual contracting offices have developed their own approaches to implementing award fee contracts which are not always consistent with the principles in the OMB guidance or between offices within these departments. Guidance from DOD, NASA, DOE, and OMB has stated that allowing contractors a second chance at unearned fees should be limited to exceptional circumstances and should require approval at a high level. Allowing contractors an opportunity to obtain previously unearned fee reduces the motivation of the incentive in the original award fee period. Three of the 5 agencies have established policies that either prohibit or limit the use of rollover. However, before changes in policies and guidance that established these limits, the use of rollover was prevalent in DOD contracts. In 2005, we reported that for DOD award-fee contracts active for fiscal years 1999 through 2003, an estimated $669 million was rolled over across all evaluation periods. In almost all of the 50 DOD contracts we reviewed, rollover is now the exception and not the rule. While in 2005, we identified that 52 percent of all DOD programs rolled over fee, only 4 percent of the programs in our sample continue this practice. We reviewed active contracts from our 2005 sample and found that the limitation on the use of rollover will save DOD more than an estimated $450 million on 8 programs from April 2006 through October 2010. In some cases, entire DOD contracting commands have strictly limited the use of rollover. One Air Force contracting officer told us that even if he wanted to roll over a portion of the unearned fee, the fee determining official (FDO) would not allow it. This change in policy has required a change in culture on both the government’s and contractor’s part. In our review of an Air Force contract for a satellite program, we found that despite receiving 0 percent of the award fee for unsatisfactory performance, a contractor sent the program a written request to include the $10 million in unearned fee in the next period. The program denied this request and has not allowed any rollover. The program ceased rolling over unearned fees to subsequent award fee periods to conform to the new policy and will save an estimated $20 million. While our analysis of DOD contracts has demonstrated the savings that can be achieved by not rolling over unearned fee, we found contracts at DOD, DOE, HHS, and DHS that continue to allow contractors second chances at unearned fees. DOD award fee letters issued as recently as January 2009 indicate that rollover is still being used. For example, in the most recent evaluation of a DOD contract for mobile radios, the program continued to recommend that funds be rolled over to subsequent periods after over $2 million in rollover fees had already been earned by the contractor. Several contracts we reviewed at other agencies allowed for 100 percent of the unearned fee to be earned in later periods. For example, in a DHS Transportation Security Administration contract for personnel services we found that a contractor that scored above average and received 86 percent of the fee in a particular period was allowed a second chance at 100 percent of the remaining fee in the next period. Additionally, an HHS Centers for Medicare and Medicaid Services award fee plan that was used on several contracts we reviewed stated that the unearned fee is placed in a separate award fee pool to be used at the discretion of the FDO. The FDO can roll over up to 100 percent of the unearned fee as long as the money is spent during the same contract year. To ensure that award fees are being used to motivate contractor performance, guidance, where available, from each of the agencies we reviewed states that award fees should be linked to acquisition outcomes such as cost, schedule, and performance. OMB’s guidance states that incentive fee contracts, which include award fee contracts, should be used to achieve specific performance objectives established prior to contract award, such as delivering products and services on time, within cost goals, and with promised performance outcomes. OMB’s guidance also states that awards must be tied to demonstrated results, as opposed to effort, in meeting or exceeding specified performance standards. Contracting officers and program managers across all five agencies we reviewed stated that a successful award fee contract should maintain a portion of fee based on a subjective evaluation of how the contractor identified and responded to issues and challenges and how it mitigated risks, but could benefit from objective targets that equate to a specific amount of fee. In August 2008, NASA’s Deputy Director noted that requirements that do not support desired outcomes should not be included in contracts and that award fees should generally only be used in complex contracts. NASA now requires that award fee contracts are accompanied by a documented cost-benefit analysis, although the requirement is too new to judge its effect. Some contracts we reviewed ensure that award fee evaluations are accurately measuring contractor performance by incorporating objective criteria to serve as inputs for the evaluation. Other contracts combined the subjective criteria of an award fee contract with the objective targets of an incentive fee contract to ensure that specific metrics are evaluated on their actual outcomes. These subjective criteria are often described as program management, cost management, or communication and allow for a broader evaluation of contractor performance. Officials that supported the use of subjective criteria noted that they must be accompanied by definitions and measurements of their own. The combination of objective and subjective measurements describes a multiple incentive contract that incorporates elements of both award and incentive fee contracts. While officials at several agencies told us that this is the preferred structure for incentivizing contractor performance and the FAR states that it is allowed, there is no guidance on how to balance or combine these contract types. OMB’s guidance states that award fees must be tied to demonstrated results, as opposed to effort, in meeting or exceeding specified performance standards. Agencies varied in the extent to which criteria used in contracts allowed them to evaluate results. For example, several DOD contracts we reviewed have included more clearly defined criteria, including the Joint Strike Fighter program that has, according to program officials, created formulas that measure software performance, warfighter capability, and cost control. The criteria, based on metrics, constitute about 30 percent of the total award fee pool. In comparing periods before and after the application of these criteria, the contractor has consistently scored lower in the performance areas than in previous periods where less defined criteria were applied. Because the program has been able to more accurately assess contractor performance, the program has saved almost $29 million in less than 2 years of the policy change. Similarly, our review of a contract for a missile defense system found that greater adherence to cost and schedule criteria prevented the program from paying $39 million for events that did not take place within specified time frames. In addition to the Joint Strike Fighter, other DOD programs that were active before the guidance was issued and not required to follow it have incorporated it voluntarily with program and contracting officials recognizing the benefits of applying the new practices. In some cases they were able to do this through unilateral changes to the award fee plan. In others, changes required negotiations with the contractor. However, in other contracts we reviewed we found criteria being used to evaluate contractor performance that had little to do with acquisition outcomes. For example, an HHS contract for call center services awarded fees based on 19 performance categories which included results based criteria, such as response times, but also included criteria based more on efforts, such as requiring the contractor to ensure that staffing levels were appropriate for forecasted volumes during hours of operation, rather than measuring results. The amount of fee established for satisfactory performance or meeting contract requirements generally awards the contractor for providing the minimum effort acceptable to the government. In our review of contracts, we found that programs used a broad range in setting the amount of fee available for satisfactory performance, but many set it at a level that left little fee to motivate excellent contractor performance. For example, DOE’s Office of Science uses a model that sets the amount of fee able to be earned for meeting expectations at 91 percent, thus leaving 9 percent to motivate performance that exceeds expectations. In contrast, in an HHS contract for management, operations, professional, technical and support services for National Institute of Allergy and Infectious Diseases animal care facilities, the contractor earns 35 percent of the award fee for satisfactory performance, leaving 65 percent of the fee to motivate excellent performance. In an effort to truly concentrate the award fee on excellent performance, one contract we reviewed for Medicare services provides no award fee for satisfactory performance. NASA’s guidance establishes satisfactory at a level that leaves 30 percent to motivate above satisfactory performance. DOD’s guidance states that satisfactory performance should earn no more than 50 percent of the available award fee. This allows the program to incentivize above satisfactory performance with the remaining 50 percent of the award fee. However, not all DOD programs have followed its guidance. For example, a Missile Defense Agency (MDA) contract signed in December 2007, awards the contractor up to 84 percent of the award fee pool for satisfactory performance, which the agency defines as meeting most of the requirements of the contract. This leaves only 16 percent of the award fee pool to motivate performance that fully meets contract requirements or is considered above satisfactory. While the scale on which the contractors are evaluated is important in determining how much fee is reserved for motivating excellent performance, the judgment of the evaluators and their interpretation of the scale also have an effect. Contracting officers we spoke with varied in their interpretation of how to use the evaluation scale. While DOD has provided guidance on defining adjectival ratings for contractor performance, some programs continue to define meeting contract requirements as excellent performance. For example, on an Air Force program contracting for support services for staff stationed overseas, a contracting official stated that the contractor “has to do a pretty bad job to receive a rating of “good”,” a rating that pays in excess of 85 percent of the award fee. The median award fee for this particular Air Force program is 100 percent over the course of 8 award fee periods over 2 contracts. These evaluations provide little motivation for improved performance despite fee determination letters that consistently noted that the contractor had room to improve. The data we collected on over 645 award fee periods in 100 contracts provided a wide range of evaluation scores, including 6 periods in which the contractor earned no fee. However, our analysis of data collected from DOE and HHS, which included all contracts over $50 million that were identified as award fee contracts from fiscal year 2004 through 2008, showed that the median award fee paid at these agencies was over 90 percent of available award fees as shown in table 4. Contractors were routinely rated at a level that reflected excellent performance. DOD’s own analysis of its use of award fees in 2007 also showed that it pays a median of 93 percent of available award fees. While our review of NASA contracts was limited to three active contracts that were reviewed in our previous work, they too had a median of 90 percent of available fees paid. The median award fee paid at DHS, also shown in table 4, was 83 percent of available fees, indicating that its contractors are typically rated lower than those at the other agencies. DOD, NASA, and OMB have promulgated guidance that no award fee should be paid for performance that does not meet contract requirements or is judged to be unsatisfactory. However, while the median award fee scores indicate satisfaction with the results of the contracts, programs across the agencies we reviewed continue to use evaluation tools that could allow for contractors to earn award fees without performing at a level that is acceptable to the government under the terms of the contract. For example, an HHS contract for maintaining a Medicare claims processing system rates contractor performance on a point scale, from 0 to 100, where the contractor can receive up to 49 percent of the fee for unsatisfactory performance, 50 to 69 percent for marginal performance, and 70 to 79 percent for satisfactory performance (defined as meeting contract requirements). Therefore, the contractor could receive up to 79 percent of the award fee for satisfactory performance, or $1.8 million over the course of the contract. Another contract for operations and technical support at the National Cancer Institute uses a scale that awards up to 59 percent of the award fee for performance that is described as failing to meet customer requirements. The same scale provides up to 79 percent of the award fee, while still not requiring the contractor to fully meet customer requirements. In the contracts we reviewed, DOE’s median award fee paid was 91 percent, indicating satisfaction with the results of the contracts. However, divisions use different approaches in evaluating contractor performance. While the evaluation tool used by NNSA does not allow for payment of award fees for unsatisfactory performance, the evaluation method used by the Office of Science allows a contractor to earn up to 84 percent of the award fee for performance that is defined as not meeting expectations. Contracting officers we spoke with defined meeting expectations differently with some stating that a contractor who performed satisfactorily would meet expectations and others requiring exceptional performance to meet their expectations. In 2007, the Office of Science eliminated use of adjectival distinctions such as “satisfactory” and “excellent” in favor of letter grades and a numerical score system to communicate performance levels and determine award fee amounts. Current Office of Science guidance tasks each site office, with assistance from headquarters, with determining the requirements and milestones for each performance measure and target. While the office has favored the new system, it has not provided instructions on defining satisfactory performance or equating letter grades to adjectival language used in the OMB guidance. Further, current award fee plans for some programs using the Office of Science lab appraisal process allow for award fee to be earned at the C level, which guidance defines as performance in which “a number of expectations ... are not met and/or a number of other deficiencies are identified” with potentially negative impacts to the lab and mission. As much as 38 percent of fee can be earned for objectives that fall in this category, according to Office of Science guidance, establishing a system that rewards below standard performance. While having an evaluation tool in place to prevent award fees from being paid for unsatisfactory performance is important, it is equally important to adhere to the tool that is used. In a Customs and Border Protection contract for maintenance of aircraft, the contractor switched to a more costly method of hazardous waste disposal to reduce its own perceived risks without communicating with the government. The evaluation described the lack of communication as questionable use of taxpayer funds for parochial interests without the coordination and consultation of government representatives. The evaluation noted that the contractor’s approach was egregious and gave the contractor the minimum score of 70, stating that eliminating the fee entirely for poor communication would ignore its performance in other areas. However, in two subsequent periods when the contractor did not respond to identified areas for improvement, the program determined the contractor’s performance to be marginal, resulting in no award fee being paid for those periods. DOD is currently the only agency required to collect data, evaluate the effectiveness of award fees, and share proven strategies in using this contract type. While DOD has collected information on award fee contracts in 2007 and 2008 in accordance with legislative requirements, these data are not being used to evaluate the effectiveness of award fee contracts. While the 2009 National Defense Authorization Act directs that the FAR be amended to require executive agencies to collect data on award fees, other agencies do not collect these data outside of individual programs. However, within certain programs, automated tools are being used to evaluate the use of award fees. Further, while OMB directed agencies to broadly disseminate its guidance and suggested that agencies find and share information on these contracts using existing web based resources, contracting officials we spoke with stated that they rely on informal networks for sharing information on the use of award fees. While programs have paid more than $6 billion in award fees over the course of the 100 contracts in our review, none of the five agencies has developed methods for evaluating the effectiveness of an award fee as a tool for improving contractor performance. Instead, program officials noted that the effectiveness of a contract is evident in the contractor’s ability to meet the overall goals of the program and respond to the priorities established for a particular award fee period. However, officials were not able to identify the extent to which successful outcomes were attributable to incentives provided by award fees versus external factors, such as maintaining a good reputation. When asked how they would respond to a requirement to evaluate the effectiveness of an award fee, officials stated that they would have difficulty developing performance measures that would be comparable across programs. Additionally, officials at NASA noted that while cost and schedule are relatively easy to measure, the government may not fully realize the effectiveness of performance until the end of a program. For example, in a satellite program, a contractor’s performance becomes meaningless without a successful launch. Of the five agencies we reviewed, DOD is the only agency that collects some type of data on award fee contracts. In 2006, legislation required DOD to develop guidance on the use of award fees that included ensuring that the department collects relevant data on award and incentive fees paid to contractors and that it has mechanisms in place to evaluate such data on a regular basis. In response to the new DOD guidance, data were collected on 576 contract actions placed under 350 contracts for which fee or incentive determinations were made during calendar year 2007. This included $2.3 billion in award and incentive fees available during the period. DOD officials told us that they have shared the analysis of these data with the Senior Procurement Executives of the military services and other Defense agencies. Additionally, the legislation required guidance to include performance measures to evaluate the effectiveness of award and incentive fees as a tool for improving contractor performance and achieving desired program outcomes. However, DOD was not able to establish metrics to evaluate the effectiveness of award fees in terms of performance. DOD pointed out that the data collected on objective efficiencies do not reflect any consideration of the circumstances that affected performance, a critical element in determining award fees. DOD, which compared fees earned to cost and schedule measurements, stated in its analysis that the metrics used to evaluate the effectiveness of the incentives included 137 actions that measured cost and schedule efficiencies. While this was 24 percent of the actions it reviewed, it represented 67 percent of the award fees paid. DOD officials noted that the data indicated that lower fees were earned when cost or schedule efficiencies were less than 90 percent. While no agency has developed a tool to track and evaluate the use of award fees, some programs we reviewed have done so individually. Citing that automation can increase the effectiveness, efficiency, transparency, and integrity of the award fee process, one MDA program has developed an automated award fee tool that allows government employees to evaluate, comment on, and offer feedback on all performance criteria. The tool also captures performance inputs and descriptions of performance standards and allows administrators to analyze user ratings to normalize and remove rating bias. While the tool is still in the stages of final testing, MDA program officials stated that the tool has provided this particular MDA program with immediate and effective results in managing the award fee process. However, this automated system has not been implemented across the agency and not all MDA program officials believe that it is beneficial. Similarly, the National Cancer Institute uses a Web-based interface that collects performance information provided by the contractor’s customers to facilitate performance assessments. Officials stated that this tool saves them numerous hours of collecting and sifting through performance data and ensures that all evaluators are making judgments based on the same materials. The guidance issued by OMB in December 2007 included instructions for broad dissemination to agency personnel who have responsibilities for the effective planning, execution, and management of acquisitions. In addition, according to an OMB official, many agencies served on an interagency working group that was created at the suggestion of the guidance. Participation on the working group was at the agency headquarters level and involved officials from each of the agencies we reviewed. The interagency working group initiated a separate working group to review and amend the FAR. However, contracting officials at offices within DOE, DOD, DHS, and HHS that develop and execute award fee guidance and practices were not specifically represented in either group, were generally not aware of either of these groups, and were not asked to provide opinions, perspectives, or experiences to either group. Recent legislation required DOD to develop guidance to provide mechanisms for sharing proven incentive strategies for the acquisition of different types of products and services among contracting and program management officials. The Defense Acquisition University (DAU) has established an online community of practice on award fees and is currently developing additional guidance for DOD on the use of award fee contracts. Within DOD, we found that information sharing on best practices and lessons learned is inconsistent between contracting commands. For example, contracting officers at one Air Force command showed us specific guidance and document templates that they received along with detailed training on using award and incentive fee contracts. However, at another Air Force command, contracting officers told us that they do not generally share strategies on using award fees and if they were to do so, it would be through informal networks. Contracting officers at DOE, DHS, and HHS also stated that they were unaware of any formal networks or resources for obtaining and sharing best practices, lessons learned, or other strategies for using award fee contracts. Instead, they rely on informal networks or existing guidance from other agencies such as DOD. Contracting officials noted that the specific nature of their missions makes it difficult to adopt the practices of other agencies. In some cases, contracting officials are taking steps to provide oversight for a number of contracts to achieve consistency and identify unsuccessful practices. For example, at MDA, NNSA, and one Air Force command, the determination of award fees is performed by a senior executive who compares the results of several contracts to ensure that a uniform evaluation process and common criteria are used when possible. Similarly, according to DOE procurement officials, at the Office of Environmental Management award fee plans are circulated among contracting officers and program managers who review them for criteria that have been successful or problematic in past contracts and at the Office of Science, award fee plans are reviewed and approved annually by headquarters. NASA has a similar process in which programs discuss their performance outcomes at a monthly meeting with the focus on one particular program. NASA officials stated that the use of award fees and the criteria being used to measure contractor performance are frequent topics in these meetings. Award fee contracts can motivate contractor performance when certain principles are applied. Linking fees to acquisition outcomes ensures that the fee being paid is directly related to the quality, timeliness, and cost of what the government is receiving. Limiting the opportunity for contractors to have a second chance at earning previously unearned fee maximizes the incentive during an award fee period. Additionally, the amount of fee earned should be commensurate with contractor performance based on evaluation factors designed to motivate excellent performance. Further, no fee should be paid for performance that is judged to be unsatisfactory or does not meet contract requirements. DOD, through revised guidance, has realized benefits from applying these practices in some of its contracts, including some that, because they were active prior to its issuance, are not required to follow the guidance. While these principles have been stated in OMB’s guidance, they have not been established fully in guidance at all five agencies we reviewed, notably DOE, DHS, and HHS. Guidance, while an important first step, will not achieve the desired effect of motivating excellent contractor performance unless it is consistently implemented. Based on our work, this guidance is not being consistently implemented. Further, the lack of methods to evaluate effectiveness and information sharing among and within agencies has created an atmosphere in which agencies are unaware of whether these contracts are being used effectively and one in which poor practices go unnoticed and positive practices are isolated. To ensure broad implementation of OMB’s guidance and positive practices in using award fees, we are making three recommendations to executive agencies. We recommend that the Secretaries of Energy, Health and Human Services, and Homeland Security update or develop implementing guidance on: developing criteria to link award fees to acquisition outcomes such as cost, schedule, and performance; using an award fee in combination with incentive fees to maximize the effectiveness of subjective and objective criteria; determining when rolling over unearned fees to subsequent periods establishing evaluation factors, including definitions of performance, associated fees, and evaluation scales, that motivate contractors toward excellent performance; and prohibiting payments of award fees for performance that is judged to be unsatisfactory or does not meet contract requirements. To promote the application of existing guidance and expand upon improvements made in using award fees, we recommend that the Secretary of Defense: in preparation for regulatory changes to the FAR , emphasize the importance of consistently adhering to current guidance for all contracts in the interim; review active contracts issued before the effective date of the 2007 guidance for opportunities to apply the guidance when efficiencies can be obtained through unilateral decisions at a minimal cost to the government; and provide guidance on using award fees in combination with incentive fees to maximize the effectiveness of subjective and objective criteria. To assist agency officials in evaluating the effectiveness of award fees, we recommend that the Secretaries of Defense, Energy, Health and Human Services, and Homeland Security, and the Administrator of the National Aeronautics and Space Administration establish an interagency working group to (1) determine how best to evaluate the effectiveness of award fees as a tool for improving contractor performance and achieving desired program outcomes and (2) develop methods for sharing information on successful strategies. We provided a draft of this report to DOD, DOE, DHS, HHS, and NASA. In commenting, each agency concurred with our recommendations. DHS and HHS noted that they have been actively engaged in a FAR working group and indicated their intention of working with that group to address our recommendation for updated guidance. DOD’s response to its specific recommendation stated that it will emphasize the importance of consistently adhering to current guidance and will advise that this guidance be applied before the effective date as opportunities allow. Additionally, each agency noted that it is a member of an interagency incentive contracting working group and proposed that this group be leveraged to facilitate implementing our recommendation on identifying methods to evaluate the effectiveness of award fees and sharing successful strategies. We agree that working through these existing groups would be an adequate approach in implementing our recommendations. DOD, DOE, DHS, and NASA provided written comments that are included as appendices IV, V, VI, and VII respectively. HHS provided oral comments on our draft. In addition, agencies provided technical comments, which we incorporated where appropriate. As agreed with your offices, unless you publicly announce its contents earlier, we plan no further distribution until 30 days from the date of this report. We then will provide copies to the Secretaries of Defense, Energy, Health and Human Services, and Homeland Security and the Acting Administrator of the National Aeronautics and Space Administration. In addition, this report will be made available at no charge on the GAO Web site at http://www.gao.gov. Please contact me at (202) 512-4841 or huttonj@gao.gov if you have any questions regarding this report. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. Key contributors to this report are acknowledged in appendix VIII. To identify the actions agencies have taken to revise or develop policies or guidance on the use of award fees, we assessed procurement policies at the departments of Defense (DOD), Energy (DOE), Health and Human Services (HHS), and Homeland Security (DHS) and the National Aeronautics and Space Administration (NASA). These five agencies provided over 95 percent of the total dollars obligated against contracts with an award fee in fiscal year 2008, according to the Federal Procurement Data System (FPDS). We reviewed our prior work on the use of award fees at DOD and NASA to identify policies and guidance in place and examined these agencies in regards to changes that were implemented based on our recommendations, legislative requirements, internal guidance, and governmentwide guidance from the Office of Management and Budget (OMB). For the other agencies, DOE, DHS, and HHS, we reviewed existing guidance on the use of award fees where available and compared it to OMB’s guidance. We interviewed procurement officials at each agency to discuss planned and implemented policy changes as they related to the OMB guidance. To determine whether current practices for using award fee contracts are consistent with OMB guidance, we reviewed data from 645 evaluation periods in 100 contracts at the five agencies from fiscal year 2004 through fiscal year 2008, allowing for a comparison of practices before and after OMB’s guidance. At DOD, we collected data on 40 active and follow-on award fee and multiple incentive type contracts used in our prior review. We also examined the 10 award fee contracts for over $10 million that were signed after the DOD guidance’s effective date of August 1, 2007 and had held at least one award fee evaluation. Where applicable, we identified the programmatic and monetary effect of implementing policy changes. We estimated cost savings at DOD achieved through the limitation of rollover of unearned fees and other changes in award fee practices consistent with 2007 DOD guidance by comparing the dollar amounts of rollover as a proportion of total available award fee pools before and after our recommendation to issue guidance on when rollover is appropriate. We also examined each program’s savings from canceling their rollover policy by projecting a reasonable dollar amount based upon historical data that they would have paid in rollover had they continued using the original policy. For award fee periods that have taken place or will take place in fiscal years 2009 and 2010, we estimated the amount of unearned fee based on historical averages. At NASA, we reviewed 3 active contracts from our prior review of 10 CPAF contracts. In our prior review, we extracted information from FPDS on the top ten dollar value NASA contracts active between fiscal years 2002 and 2004 that were coded as CPAF. At DOE, DHS, and HHS, we collected data on 47 contracts that represent the universe of CPAF, fixed-price-award-fee, and multiple incentive type contracts with an award fee component that had obligations greater than $50 million from fiscal year 2004 through fiscal year 2008. To ensure the validity of the database from which we drew our contracts, we confirmed the contract type of each of the 47 contracts we selected through DOE, DHS, and HHS contracting officers and contract documentation. Contracts in our sample conducted at least one award fee period between fiscal years 2004 and 2008 and issued a letter of notification (fee determination letter) to the contractor regarding at least one award fee payment. For each of the 100 award fee contracts in our sample of the five agencies, we collected four primary data points for each evaluation period: (1) the award fee available, (2) the award fee paid,(3) the amount of unearned fee rolled over into subsequent evaluation periods, and (4) the end date of the award fee period. In most cases, contracting and program officials submitted the data from firsthand documentation such as award fee plans, contract modifications, and fee determining official letters. From these data, we calculated the percentage of the available fee that was awarded for individual evaluation periods, entire contracts to date, and the overall sample. We collected data from agencies within the five departments and met with selected procurement, contracting, and program officials to obtain the perspective of users of award fees. At these meetings we discussed experiences, policies, and guidance related to use of the award fees. Agencies from which we collected data include: Air Force Space and Missile Systems Center Air Force Materiel Command Air Force Aeronautical Systems Center Air Force Security Assistance Center Air Force Logistics Command Air Force Space Command Army Chemical Materials Agency White Sands Missile Range Fort Polk Army Reserves Army Space and Missile Defense Command Naval Air Systems Command Naval Sea Systems Command Space and Naval Warfare Systems Center National Nuclear Security Administration Office of Civilian Radioactive Waste Management Office of Legacy Management Office of Environmental Management Office of Health, Safety and Security Office of Science Strategic Petroleum Reserve Department of Health and Human Services: Agency for Healthcare Research and Quality Centers for Disease Control and Prevention Centers for Medicare and Medicaid Services National Institutes of Health Substance Abuse and Mental Health Services Administration Department of Homeland Security: Customs and Border Protection Federal Emergency Management Agency Transportation Security Administration U.S. Coast Guard We conducted this performance audit from August 2008 through May 2009 in accordance with generally accepted government auditing standards. These standards require that we plan and perform the audit to obtain sufficient and appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. Award fee: An amount of money added to a contract, which a contractor may earn in whole or in part by meeting or exceeding the criteria stated in the award fee plan. These criteria typically relate to subjective areas within quality, critical processes, technical ingenuity, cost-effective management, program management, subcontract management, and other areas that may have unquantifiable behaviors. Award fee plan: A document that captures the award fee strategy. The plan details the procedures for implementing the award fee by structuring the methodology of evaluating the contractor’s performance during each evaluation period. Award fee pool: The total of the available award fee for each evaluation period and base fee (if applicable) for the life of the contract. Award fee review board (AFRB): The AFRB evaluates the contractor’s overall performance for the evaluation period in accordance with the Award Fee Plan. The board is comprised of Government personnel only whose experience in acquisition allows them to analyze and evaluate the contractor’s overall performance. Base fee: An award-fee contract mechanism that is an amount of money over the estimated costs (typically in the range of 0 to 3 percent of the contract value), which is fixed at the inception of the contract and paid to the contractor regardless of performance in a cost-plus-award-fee contract. A base fee is similar to the fixed fee paid to a contractor under a cost-plus-fixed- fee contract that also does not vary for performance. Cost contract: A cost-reimbursement contract in which the contractor receives no fee. A cost contract may be appropriate for research and development work, particularly with nonprofit educational institutions or other nonprofit organizations, and for facilities contracts. Cost-plus-award-fee contract: A cost-reimbursement contract that provides for a fee consisting of a base amount (which may be zero) fixed at inception of the contract and an award amount, based upon a judgmental evaluation by the government, sufficient to provide motivation for excellence in contract performance. Cost-plus-incentive-fee contract: A cost-reimbursement contract that provides for an initially negotiated fee to be adjusted later by a formula that objectively measures the performance of the contractor. Cost-reimbursable contract: A contract that provides for payment of the contractor’s allowable cost to the extent prescribed in the contract not to exceed a ceiling. Evaluation criteria: The criteria that are used to grade each category of performance. The criteria should emphasize the most important aspects of the program to facilitate the contractor doing its utmost to deliver outstanding performance. The criteria should be specific to the program and clearly stated in the contract. Evaluation period: The period of time upon which an award fee is based. This can be a specific increment of time (one year) or based upon the completion of an event (preliminary design review). An award fee amount is tied to each period of time or each event and the award fee board determines the appropriate fee for this period of time subject to approval by the fee determining official. Fee determining official (FDO): The FDO makes the final determination regarding the amount of award fee earned during the evaluation period by the contractor. Fixed-price contract: A contract that provides for a price that is either fixed or subject to adjustment obligating the contractor to complete work according to terms and for the government to pay the specified price regardless of the contractor’s cost of performance Fixed-price-award-fee contract: A variation of the fixed-price contract in which the contractor is paid the fixed price and may be paid a subjectively determined award fee based on periodic evaluation of the contractor’s performance. Fixed-price incentive contract: A fixed-price contract that provides for adjusting profit and establishing the final contract price by application of a formula based on the relationship of total final negotiated cost to total target cost. Incentive contract: A contract used to motivate a contractor to provide supplies or services at lower costs and, in certain instances, with improved delivery or technical performance, by relating the amount of fee to contractor performance. Multiple incentive contract: A contract which contains both incentive and award fee criteria. This type of contract could be coded as a combination contract in the Federal Procurement Data System (FPDS). Provisional award fee payment: A payment made within an evaluation period prior to a final evaluation for that period. This payment is subject to restrictions and must be paid back to the government if the award fee board decides that this money was not earned. Reallocation: The process by which the Government moves a portion of the available award fee from one evaluation period to another for reasons such as Government-caused delays, special emphasis areas, and changes to the Performance Work Statement (PWS). Rollover: The process of transferring unearned available award fee from one evaluation period to a subsequent evaluation period, thus allowing the contractor an additional opportunity to earn that unearned award fee. Appendix III: OMB Guidance on the Use of Award and Incentive Fee Contracts planning required to implement an incentive type contract and the amount of additional resources required for monitoring and determining awards. Risk and cost analyses related to the use of award and incentive contracts should be prepared in writing and approved at a level above the contracting officer or as determined by the agency. Incentive fees must be predetermined in writing and processes for awarding the fees must be included or cross-referenced in the acquisition plan (see FAR 7.105(b)(4)(i)). This incentive fee plan should include standards for evaluating contractor performance and appropriate incentive fee amounts. When considering the incentive fee arrangement, the plan should distinguish between earning potential for satisfactory versus excellent performance. Metrics should clearly describe what is required and at what point a contractor is considered successful. Additionally, agencies should develop guidance on when it is appropriate to award rollovers of unearned fee to a subsequent evaluation period. Rolling over fees is not the preferred method for incentivizing the contractor to perform above satisfactorily and should be permitted on a limited basis and require prior approval of the appropriate agency official. Using the attachment as a guide, Chief Acquisition Officers should review and update existing agency guidance on incentive fee contracting practices to ensure that fees are awarded in accordance with current regulations and that the guidance addresses the concerns of this memorandum. In addition, during an agency’s internal audit process, incentive fee contracts should be reviewed as part of the program management review process. Information on how well incentive fees are achieving their intended purpose and other related lessons learned can be found and shared on the Acquisition Community Connection on https://acc.dau.mil/CommunityBrowser.aspx?id=105550&lang=en-US. To help develop best practices, guidance, and templates, OFPP requests that agencies identify an incentive and award fee point of contact. These individuals may be asked to contribute examples and lessons learned to an interagency working group or to assist in communication and awareness efforts. Please submit the person’s name, title, telephone number, and e-mail address to Susan Truslow at OFPP by January 7, 2008. Please ensure broad dissemination of this memorandum among agency personnel who have responsibilities for the effective planning, execution, and management of your acquisitions. Questions may be referred to Susan Truslow at (202) 395-6810 or struslow@omb.eop.gov or Pat Corrigan at (202) 395-6805 or pcorrigan@omb.eop.gov . Thank you for your attention to this important matter. cc: Chief Information Officers Consult agency policy and guidance that supplement FAR 16.4, Incentive Contracts. Ensure market research documentation and the acquisition plan sufficiently state desired outcomes, performance requirements, milestones, risks and cost benefits associated with choice of contract type (FAR 7.105). Conduct and document risk and cost/benefit analyses that support use of an incentive type o Conduct a risk assessment and ensure incentive strategies are consistent with the level of risk assumed by the contractor and motivate the contractor by balancing awards with negative consequences; o Determine whether administrative costs associated with managing the incentive fee are outweighed by the expected benefits; and o Ensure sufficient human resources are available to properly structure and monitor the contract. Ensure evaluation factors are: o Meaningful and measurable; o Directly linked to cost, schedule, and performance results; and o Designed to motivate excellence in contractor performance by making clear distinctions in possible award earnings between satisfactory and excellent performance. Ensure the incentive fee plan: o Defines clearly the standards of performance for each rating category (e.g., satisfactory, above satisfactory, excellent); o Defines clearly the percentage of fee the contractor should be paid for each of these o Documents roles and responsibilities for those involved in monitoring contractor performance and determining award fees; o Provides detailed guidance on steps in the evaluation process for agency o Establishes a base fee. Good business practice allows the contractor more than 0% for base fee. This way, the award fee promotes above average performance; and o Obtains appropriate approval in accordance with agency policy. Ensure rollover fees are allowed only in limited circumstances in accordance with agency policy. The following is GAO’s comment on the Department of Homeland Security’s letter dated May 28, 2009. While we agree DHS has taken several steps to improve the use of award fee contracts since the issuance of OFPP’s guidance, DHS’s changes to the Homeland Security Acquisition Manual do not fully address the issues in the OFPP guidance. As we point out in our report, the manual does not describe standards or definitions for determining various levels of performance nor does it address issues related to rollover. In addition to the individual named above, Thomas Denomme, Assistant Director; Ann Calvaresi Barr; Laurier Fish; Kevin Heinz; Julia Kennon; Farhanaz Kermalli; John Krump; Caryn E. Kuebler; Karen Sloan; and Monique Williams made key contributions to this report.

Summary: In prior work, GAO found that contractors were paid billions of dollars in award fees regardless of acquisition outcomes. In December 2007, the Office of Management and Budget (OMB) issued guidance aimed at improving the use of award fee contracts. GAO was asked to (1) identify agencies' actions to revise or develop award fee policies and guidance to reflect OMB guidance, (2) assess the consistency of current practices with the new guidance, and (3) determine the extent agencies are collecting, analyzing, and sharing information on award fees. GAO reviewed the Departments of defense (DOD), Energy (DOE), Health and Human Services (HHS), and Homeland Security (DHS) and the National Aeronautics and Space Administration (NASA)--agencies that constituted over 95 percent of the dollars spent on award fee contracts in fiscal year 2008. From fiscal year 2004 through fiscal year 2008, agencies have spent over $300 billion on contracts that include monetary incentives, or award fees, for performance that is evaluated against subjective criteria. OMB's guidance on using award fees includes principles such as limiting the opportunities for earning unearned fees in subsequent periods, linking award fees to acquisition outcomes, designing evaluation criteria to motivate excellent performance, and not paying for performance that is unsatisfactory. These principles are largely reflected in DOD's and NASA's updated guidance on the use of award fees. For example, DOD now prohibits payment of award fees for unsatisfactory performance, and NASA requires a documented cost-benefit analysis to support the use of an award fee contract. However, DOE, DHS, and HHS vary in the extent to which their agency-wide guidance reflects the OMB guidance. These departments generally rely on operational divisions to develop award fee guidance; however, many acquisition professionals at these agencies were unaware of the contents of the OMB guidance. Current practices for using award fee contracts at agencies GAO reviewed often are inconsistent with the new guidance. However, where the revised policies have been applied, the results have been hundreds of millions of dollars in cost savings and better use of government funds. For example, by limiting second chances at unearned fees in eight programs, GAO estimates that DOD will save over $450 million through fiscal year 2010. These practices, however, are not being implemented across DOD. NASA programs now document cost benefit analyses to justify using award fee contracts. Without clear guidance, agencies within DOE, HHS, and DHS have developed various approaches to using award fees. For example, while DOE's median award fee paid indicates satisfaction with the results of its contracts, its Office of Science uses a scoring system that could allow for payment of up to 84 percent of an award for performance that does not meet expectations. Most of the agencies we reviewed continue to allow contractors second chances at unearned fees. For example, at DHS, a contractor was able to earn 100 percent of its unearned fee in a subsequent period. Agencies do not always use criteria that are based on measuring results. For example, one HHS contract for a call center included criteria that focused more on efforts, such as maintaining proper staffing levels during hours of operation, rather than on measuring results. Only DOD collects data on the use of award fees. However, the data are largely used to respond to legislative requirements for award fee information. Agencies generally do not have methods to evaluate the effectiveness of award fees. While individual programs and some offices have taken steps to evaluate award fee criteria, officials stated that identifying metrics to compare performance across programs would be difficult. Further, while GAO found effective practices within some agencies, the lack of a governmentwide or, with the exception of DOD, agencywide forum to share information allows these to remain isolated examples of potential best practices.