To determine the soundness of IRS’s PDC study as primary support for IRS’s decision to discontinue contracting out tax debt collection, we reviewed the study report and supporting documents and other data. We interviewed IRS officials and contractors involved in the study. We also reviewed the report IRS commissioned to validate the PDC study conclusion and interviewed contractor staff involved in that effort. We also interviewed officials from one of the PCA firms IRS contracted with to obtain their views on the PDC study. We reviewed and summarized program analysis guidance from various sources in developing our criteria. Such guidance came from Office of Management and Budget (OMB) Circular A-94, a previous GAO publication on evaluating federal programs, an academic research paper on costs and benefits that should be considered in making decisions on resources for tax enforcement programs, and accepted quantitative analysis criteria on sampling cases and projecting results. We compared IRS’s study methodology and report to the criteria from the various guidance. We also reviewed IRS’s Internal Revenue Manual (IRM) and interviewed officials to determine if IRS had guidance on whether and how to conduct and document economic analyses to support decisions to initiate, renew, or expand programs. We also reviewed the types of costs and benefits that IRS included in the PDC study. To determine what changes IRS has planned or made to its collection approach based on its PCA experience and the PDC study, we reviewed program documents and interviewed IRS officials on their processes and procedures for collecting tax debt. We also compared IRS’s plans for studying whether to work on PCA-type cases in the future to guidance in our previous publication on evaluating federal programs and in our report reviewing IRS’s study of the earned income tax credit. We conducted this performance audit from November 2009 to September 2010 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. In developing the PDC program, IRS officials said that contracting with PCAs was needed because Congress was unlikely to provide IRS sufficient staff to attempt collection on the inventory of cases with a lower priority. Also, officials said that PCAs would pursue cases that IRS staff would not because IRS had other higher-priority collection cases. As IRS planned the PDC program, we issued two reports with findings and recommendations to improve the program. In May 2004, we reported that IRS’s PDC study approach—comparing PCA and IRS performance for the same type of simpler cases that would be sent to PCAs—would provide limited information to judge whether using PCAs is the best use of resources. In sum, the approach conflicted with IRS officials’ position that these simpler cases would not be worked by IRS employees given the higher-priority cases in IRS’s workload. We recommended instead that IRS compare the use of PCAs to a collection strategy that officials determine to be the most effective and efficient overall way of achieving collection goals. In September 2006, we reported that IRS’s planned PDC study approach would not meet the intent of our 2004 recommendation. Because the study would not count the fees paid to PCAs as program costs, it would not compare the results of using PCAs with the results IRS could get if it was given the same amount of resources, including the fees to be paid to the PCAs. We recommended instead that IRS ensure that the study methodology and the reports on the study include the full costs of the PDC program, including the fees paid to PCAs and the best use of those federal funds. IRS agreed with the recommendations in both reports, and IRS’s actions to implement them generally are the topic of this report. IRS’s PDC study did not meet the intent of our 2004 recommendation because it did not compare using PCAs to what IRS officials determined to be the most efficient and effective overall strategy. Although IRS met part of our 2006 recommendation in that the PDC study included fees paid to PCAs as program costs, the PDC study did not fully implement our recommendation to include the best use of federal funds because, as discussed further below, the study had methodological errors and a narrow scope. IRS’s PDC study report results were released on March 5, 2009, along with IRS’s announcement that it was ending the program. The study tracked selected cases assigned to IRS’s Automated Collection System (ACS) versus PCAs and measured them in terms of cost per dollar collected, percentage of balance due collected, and percentage of cases in payment status at the end of the study period. For both PCAs and IRS, the PDC study counted the dollars collected from full payments or estimated dollars collected through installment agreements. The study results, summarized in table 1, showed that IRS performed better in each of the measures. On February 3, 2009, legislation was introduced in the House to amend the Internal Revenue Code of 1986 to repeal the authority of the Secretary of the Treasury to enter into PDC contracts. The last major action on the legislation was on February 3, 2009, when it was referred to the House Committee on Ways and Means. The Omnibus Appropriations Act of 2009, enacted March 11, 2009, prevented IRS from using fiscal year 2009 appropriated funds “to enter into, renew, extend, administer, implement, enforce, or provide oversight of any qualified tax collection contract (as defined in section 6306 of the Internal Revenue Code of 1986).” Absent the access to appropriated funds, IRS funded the administrative costs of the PDC program through its user fees until IRS was able to end all PDC activities. The Consolidated Appropriations Act of 2010 placed the same prohibition on IRS’s use of fiscal year 2010 funds. According to IRS officials, the PDC study was not originally intended or designed to be primary support for IRS’s PDC program decision. Even though other factors, such as potential increases in IRS collection staffing, were considered, based on our interviews with IRS officials and IRS’s announcement of the program’s termination, the study results played a primary role in supporting the decision. IRS officials noted that the difference in cost per dollar collected using IRS staff and PCAs was so pronounced that in their view, additional analyses would have been unlikely to change the decision that was made. Nevertheless, neither we nor IRS officials know whether the PDC study results and decision on the program would have differed significantly if the study had been designed to be primary support for IRS’s PDC program. However, errors in the study sampling methodology and the study’s narrow scope limit its usefulness in supporting the PDC decision. IRS does not have guidance on whether and how to conduct and document economic analyses to support decisions to initiate, renew, or expand programs. The IRM, which includes authoritative guidance for IRS managers, does not include guidance on economic analyses that should support decisions to initiate, renew, or expand a program. Because IRS did not perform certain analyses and documentation is not available to do those analyses, it is unclear whether the study’s results are accurate. Guidance for statistical surveys requires that a study’s sampling procedures be sufficiently described such that policymakers can assess whether the study’s results can be generalized to the settings and times of interest. The PDC study documentation mentioned a design that could represent the full population of PCA-type cases based on sample cases drawn from various groupings of PCA-type cases. However, IRS did not generalize the study results back to the full PCA case population, limiting its analysis to the sample cases. It is common for study results to differ significantly between results for the sampled cases and results when generalized back to the full population. IRS did not retain sufficient documentation on the sample selection and analysis to enable it or others at this point to calculate estimates for the population, measure the margin of error, or otherwise ensure the soundness of the sample approach. Even though IRS documentation referred to a sampling approach, IRS officials said the study was not intended to provide generalizable results. The sample cases assigned to the PCAs and IRS for comparative purposes may or may not have been similar. IRS was concerned about potential differences between the groups and tested the difference in the average balance due amount between the cases to be assigned to PCAs and IRS. The test did not find a difference. However, this was a limited test of only one variable. IRS could have tested other variables, such as taxpayer filing history, adjusted gross income, filing status, geographic region, and type of forms completed by the taxpayer. Differences in any of these variables could affect taxpayer compliance and payment. Performance differences between the PCAs and IRS could be due to differences in the composition of the groups. Because the PDC study had a narrow objective of comparing the results of collection efforts by IRS and PCAs for the PCA-type cases, its design did not consider other factors included in federal and other guidance on conducting program analyses. An objective more directly focused on providing information to serve as primary support for IRS’s decision on the PDC program could have led IRS to conduct additional analyses relevant to the decision. The findings from those analyses could have affected the program decision. Guidance on conducting program analysis to determine whether to continue a program states that the rationale for the government program being examined should be clearly stated in an analysis. Clearly stating the rationale for the program can help ensure that the analysis includes relevant factors and outcome measures, particularly if facts related to the rationale have changed and therefore need to be included in the analysis. Stating the rationale of the program and conducting related analyses can help ensure that variables measured in the study answer the questions decision makers have about the basic reasons the program exists, the continuing need for it, and the costs and benefits of continuing, revising, or ending it. For instance, one rationale for starting the program was that Congress was unlikely to provide funds for staff to work these cases. The PDC study did not cover whether this had changed and, if so, whether IRS would use any portion of additional staffing for PCA-type cases. As discussed later, the number of staff who would normally work PCA-type cases has increased in recent years. To inform a decision on whether to continue a program and ensure the best use of federal resources, government officials need information on the range of alternatives for achieving a program’s objectives at the lowest costs. Guidance states that program analyses should include alternative means of achieving program objectives by examining different program scales, methods of provision, and degrees of government involvement. The PDC study included three alternatives: (1) PCAs attempting to collect unpaid tax debts for the inventory assigned to them, (2) IRS taking collection action on the types of cases that were assigned to PCAs, and (3) IRS taking collection action on another type of inventory normally not worked. However, beyond not addressing whether to continue the PDC program, the study did not analyze alternatives for program scale, such as expanding the PDC program or scaling it back to a segment of cases that might be more cost effective for PCAs to work than IRS. For example, in deciding whether to scale back rather than eliminate the PDC program, IRS could have analyzed the types of cases, if any, where PDC performance was better than IRS performance. Assuming that this analysis pointed to favorable types of cases for PCAs, IRS then could have limited the program to working those cases, unless the benefit of doing this was so limited that running a separate PDC program with fixed costs would not be cost beneficial. According to IRS officials, IRS had effectively scaled back the program during its course because after the initial wave of cases assigned to PCAs, IRS had difficulties in obtaining a sufficient volume of cases appropriate for PCAs to work. In addition, IRS did not compare the PDC program to what it judged to be the best overall strategy for improving tax collections, as we had recommended IRS do in our 2004 report, even though the Commissioner of Internal Revenue agreed with our recommendation. In 2004, we concluded that IRS should do a study in line with federal guidance, such as comparing the results of using PCAs to the results from using the same amount of funds to be paid to PCAs in an unconstrained manner that IRS determined to be the most effective overall way of achieving its collection goals. After our report, Congress authorized the PDC program and included the provision making 25 percent of PDC collections available to IRS. Conceptually, Congress effectively made available funds for the PDC program totaling the amounts paid to PCAs plus the up to 25 percent of PDC collections that IRS could use for enforcement purposes. We continue to believe that a comparison of results of the PDC program—as authorized by Congress—to results IRS would achieve if given the same funds to use in what it judged to be the best possible manner would have better supported a decision on the PDC program. To inform a decision on whether to continue a program, government officials need complete and reliable information on all the program’s benefits and costs. Guidance for doing economic analyses states that to the extent possible all benefits and costs should be monetized to provide a standard unit of comparison. If it is not feasible to assign monetary values, other quantification of costs and benefits should be done. If quantification is not possible, at a minimum, analyses should include a comprehensive listing of the different types of benefits and costs to identify the full range of program effects. Furthermore, analyses should be explicit about the underlying assumptions used to estimate future benefits and costs. The PDC study measured the government’s cost per dollar of direct tax revenue collected, percentage of balance due collected, and percentage of cases in payment status at the end of the study period. As cited in IRS’s announcement of the decision to not renew PCAs’ contracts, the study showed that when working the same types of cases as PCAs, IRS had better results than PCAs. Specifically, IRS’s cost was $0.07 to collect a dollar itself while the government’s cost using PCAs was $0.24 for each dollar collected. According to IRS officials, IRS’s results on the cases it worked were comparable to or better than the results of working its normal cases. In addition to direct revenues, the study did not estimate or otherwise discuss indirect revenue (for example, determine whether other taxpayers are more or less likely to pay their due taxes when IRS works the cases than when PCAs do). Indirect revenues are difficult to estimate, but the study could have explored whether any logical reasons exist to indicate that indirect revenue would have varied based on which party worked the PCA-type cases. As discussed further below, the PDC study also did not include potentially important program costs and benefits for a tax enforcement program, such as costs to taxpayers to comply with requirements to pay their delinquent taxes (are costs to taxpayers lower under IRS or PCAs); equity (is IRS or are PCAs more effective in collecting taxes from taxpayers in different income groups, for example); and economic efficiency (is IRS or are PCAs more effective at working cases involving taxpayers in different industries, for example, resulting in more or less distortion of activity across types of businesses). IRS officials said that these benefits and costs were not addressed in the study because they are difficult to measure. However, IRS could have followed OMB guidance to, at a minimum, list and discuss these omitted costs and benefits in the final report. As a result, the decision to end the program rests, in part, on the assumption that important differences do not exist between IRS and PCA handling of these cases related to these costs and benefits. If IRS assumed that no such differences existed, this should have been stated in the report with the supporting rationale. Some important differences might exist. For example, it is not clear that taxpayer costs would be the same regardless of which party worked on collecting their tax debt. Taxpayer compliance costs could be lower when IRS collects taxes because 90 percent of the cases IRS worked were eligible for systemic actions, such as systemic levies of taxpayer’s assets. Such actions might incur little cost for the taxpayer beyond reading IRS’s notification of intent to levy assets, as compared to a PCA case, which required that a taxpayer first make a payment or answer or return a phone call in response to a PCA’s letter or call. Policymakers do not know whether costs to taxpayers differed between IRS and PCA collections because IRS did not consider this and other circumstances that could affect taxpayers’ costs. The study also did not include a type of collected revenue—which IRS called noncommissionable revenue—that IRS officials tracked in a measure to assess the PDC program’s performance and, in some cases, might have gone uncollected otherwise. For example, IRS did not pay PCAs a commission when debt was collected within a 10-day window after being assigned to a PCA. In establishing the program, an IRS official told us that IRS expected that many dollars would be collected within this 10- day window because taxpayers would send in payment after receiving notification that their debts were being assigned to a PCA. To the extent this occurred, not taking these revenues into account in the study may have led to an underestimate of the collections attributable to the PDC program. Other noncommissionable revenue included collections through actions IRS systemically took regardless of whether a case was assigned to a PCA, such as refund offsets. According to IRS officials, such revenues were not included in the PDC study for either PCAs or IRS collections. IRS data through fiscal year 2007 show that total collections for the PDC program were about $32.1 million with $7 million (22 percent) of the total being noncommissionable; for fiscal year 2008, the total was $37.3 million with $9.6 million (26 percent) noncommissionable; and for fiscal year 2009, the total was $28.8 million with $11.5 million (40 percent) noncommissionable. IRS’s PDC study focused on just a few types of government costs. Those costs included IRS’s costs as well as the commissions paid to PCAs. Table 2 identifies these types of costs. IRS used a variable costing methodology rather than a full costing approach to compare the costs of PCAs and IRS working inventory that they determined to be similar. According to IRS, the variable costing approach included expenses that would vary depending on an increase or decrease in the inventory assigned to either the PCAs or IRS. The study did not include previous costs, such as costs in setting up the PDC program. In general, this overall approach to including costs is consistent with OMB guidance. That guidance says that analyses should be based on incremental costs and benefits and sunk costs should be ignored. IRS did not include management oversight or information technology since such costs would not vary with the volume of cases handled. For example, IRS did not include the costs of a PDC program oversight unit that among other things was responsible for monitoring PCA performance. Although these costs likely would not vary greatly with changes in the volume of cases handled by IRS or PCAs, if the absolute costs were significantly greater either for IRS or PCAs, the difference may have affected the cost comparison, especially over relatively small numbers of cases. IRS did not document whether these costs varied significantly based on who handled the cases. To determine the cost for the PCAs, IRS used historical cost data of the PDC program for fiscal year 2008 that were generated from IRS’s Integrated Financial System. IRS apportioned those costs to the number of cases in its study. Unlike the PCAs cost data, the IRS costs were not system generated but were calculated based on estimates and assumptions. For example, IRS identified the transactions (such as phone calls made or notices sent to taxpayers) taken on each case in the sample by manually reviewing individual data posted in the taxpayers’ case files in the Integrated Data Retrieval System. To then calculate direct labor hours per case, IRS used information from inventory handling time reports and call handling time reports. IRS determined an average minutes per transaction, which it applied to the sampled cases in the study. Because the PCA-type cases were at least somewhat different than cases ACS normally handled, applying the average minutes from typical transactions may or may not have accurately reflected the time actually taken on the PCA-type cases. Other estimated costs were for telephone calls and postage costs for mailing a single installment agreement notice to taxpayers. Another fundamental program analysis principle is to estimate expected results of the program. Because decisions about the future of a program depend on the expected future results of it or any alternatives that are under consideration, future results are to be included on a discounted basis in either benefit-cost analysis (the value of expected net benefits, i.e., benefits minus costs) or cost-effectiveness analysis (to determine which alternative has the lowest costs for a given amount of benefits or which has the greatest benefits for a given amount of costs). Past performance can be relevant in helping to estimate the value of future results. IRS’s PDC study included no estimates of future costs and revenues for the alternatives studied. The study presented data and related ratios, including costs per dollar collected, only for cases worked in the past. Although the study text states that the cases studied were like those PCAs worked during the program and would work in the future, it provided no analytical basis for this assumption. Documenting any analyses of future costs and revenues was especially relevant for any decision on the future of the PDC program because IRS had previously revised the criteria for selecting PDC cases several times in order to provide the volume of work to PCAs that had been anticipated. Accordingly, reasonable questions existed about whether IRS would be able to continue providing a sufficient stream of work to PCAs and how that might affect the scope of the program and the nature of the cases PCAs would work. Absent any such documented analysis, decision makers and those overseeing the agency had a limited basis on which to assess whether PCAs likely would have been as effective at working the cases IRS could deliver in the future as they had been in working past cases. IRS officials said that they considered future results, but only outside the study, in deciding to end the PDC program. IRS officials said that during management meetings on deciding whether to continue the PDC program, they considered past program performance compared to expectations and how that might affect future performance. For example, they said that they considered gaps between IRS’s original expectations, what the program actually realized, and what the program would likely achieve in the future given actual program experience. Among other things, IRS officials noted the declining available case inventory levels for the program. They said that the program’s actual performance compared to expectations was a reason that IRS ended the program. IRS was unable to provide documentation of any analyses of future expected costs and results. Without such analyses, it is unclear to what extent the program did not meet expectations, whether IRS determined the underlying reason, and whether the program’s future performance could have been improved in a manner that could have affected the decision on the program. In authorizing the PDC program in 2004, Congress required IRS to create a measurement plan to capture information on successful collection techniques used by the contractors that IRS could adopt. The PCAs’ best practices were to be compared with IRS’s collection practices. IRS was to report on this measurement plan and its results in a mandated biennial report that was to include specific types of information. In an unpublished draft biennial report, IRS said it reviewed PCA best practices and concluded that none of them were sufficiently better than IRS’s practice to merit adoption. IRS officials provided us a draft version of the biennial report for 2007. IRS neither finalized the report nor released it. IRS officials pointed to significant transition in the PDC office during this time, as well as transitions in the Deputy Commissioner’s and the Commissioner’s offices. This draft report described IRS’s steps to identify lessons learned from PCAs. It said that IRS did not find any immediate opportunities to adopt PCA practices, but provided no details beyond this sentence. The draft report also said IRS would continue to try to identify lessons learned. IRS officials said that IRS had not changed its criteria to start regularly selecting PCA-type cases to work because the PDC study results were not sufficient to identify which of the PCA-type cases could be productively worked. However, IRS officials said that they were surprised by the study results, which indicated that IRS staff might have better results working these cases than some of the cases IRS normally works. IRS officials said that the types of cases sent to PCAs previously had been considered low priority because of low potential collection return. In establishing the PDC program, IRS officials indicated that the PCA contractors would be working inactive collection cases that IRS collection staff would not be working. For example, in May 2007, after IRS had sent the initial inventory of cases to PCAs to work, the Acting Commissioner of Internal Revenue testified that if the money invested in the PDC program was not used for it, IRS would not use the funds to work cases that would have been assigned to PCAs. Rather, the funds would be used to work other cases considered higher priority in the tax debt inventory. Because of the surprising result, IRS began a pilot study of the types of cases that had been worked by PCAs. According to IRS officials, the pilot study’s goals were to (1) provide coverage for a segment of the unpaid tax debt inventory that would not be worked because of the end of the PCA contract and (2) provide information on which types of cases IRS could fruitfully work in the future. IRS placed approximately 19,000 pilot cases with ACS over a 4-month period beginning in September 2009. IRS officials said that the agency randomly drew the cases from the database of cases that IRS had used to assign cases to the PCA staff. IRS employees cannot identify which of their assignments are PCA-type cases. As of July 2010, the status of these cases was as follows: 8,559 cases (45 percent) had been closed, for example, through some form of payment or other action; 4,954 cases (26 percent) were being worked; and 5,552 cases (29 percent) had yet to be worked. As of June 2010, IRS officials said that they expected that the work on these pilot study cases would be finished by December 2010. For these cases, IRS collected data on its activities (such as liens and levies) and the results (such as dispositions and dollars collected). Of the 8,559 cases closed as of July 1, 2010, IRS had received payment in full in 931 cases (11 percent) and had entered into installment agreements in 2,878 cases (34 percent). IRS officials said they had planned to use the collected data to determine if changes should be made to the case selection criteria to assign certain types of PCA-type cases to the IRS active collection inventory. According to IRS officials, after the pilot cases were worked, IRS’s Office of Program Evaluation and Risk Analysis (OPERA) was to study the collected data to see what happened with the cases. IRS expected that the OPERA study would take a year to complete (i.e., until about December 2011). IRS officials said that the agency was then to determine whether more PCA- type cases should be routinely worked in IRS and, if so, which ones. Following guidance on conducting program analyses helps ensure that relevant information is considered and a sound methodology is used, including stated study objectives and a clear and appropriate study design for answering the objectives. As we have reported in the past, these principles are essential when developing an analysis or evaluation plan. For example, this plan should not only be written but have sufficient detail to answer questions such as the following: Are the test goals and objectives of IRS’s pilot study clearly stated? Has IRS identified the types of data to be collected? What specific types of analysis will be performed? The documentation IRS had available describing the pilot study—a draft plan prepared in December 2009—addressed summarizing and providing data on the sampled cases (such as collection action taken, e.g., liens and levies) and the results (such as the disposition or current status of the cases and the dollars collected). But as of June 24, 2010, IRS had not produced an approved plan on how it intended to analyze the pilot study data even though it had worked about three-fourths of the cases selected for the pilot. According to IRS officials, progress in developing an analysis plan for the pilot study had been delayed by efforts to implement Consolidated Decision Analytics (CDA). Among other things, CDA is to implement new models to better predict the collection potential of unpaid tax debt cases to help ensure the best use of IRS’s collection resources. Officials said that any plan for analyzing the pilot study’s data would need to ensure that the pilot study results will be useful given IRS’s plans to more fully implement CDA in January 2011. Beyond not knowing how the pilot study data will be analyzed, IRS had not clarified its criteria on how it would use the results of any analyses to make a decision about assigning more PCA-type cases to be worked in IRS. Specifically, IRS had no documentation on the criteria that officials would use in making this decision and what factors, if any, beyond the analyses of the pilot study data would contribute to those criteria or that decision. Developing criteria would be important to ensure that the variables measured in the study would be useful in supporting the decision on whether to change case selection criteria to regularly pursue such cases. As we concluded our review, the status of the pilot study and whether or how PCA-type cases would be included in active collection case inventory became less clear. IRS provided conflicting information about determining whether PCA-type cases have sufficient collection potential to be included in its collection inventory. On one hand, officials said that CDA models were built by tracking the characteristics and collection potential of actual collection cases. On the other hand, PCA-type cases generally have not been worked by collection staff, which is why IRS began its pilot study of PCA-type cases to determine their collection potential. In its comments on our draft report, IRS said that the CDA models that would be implemented in January 2011 had overtaken the need to complete the pilot study. Examining the CDA models was outside the scope of our review. Therefore, we had not reviewed documentation on what types of data were used in developing the CDA models. However, in response to our direct question about whether PCA-type case results were used in developing the CDA models to be implemented in January 2011, an IRS official said that they had not been used. IRS’s March 2009 announcement ending the PDC program said that IRS anticipated hiring more collection staff in fiscal year 2009 and referred to support from the administration and Congress for increased IRS enforcement resources. According to IRS officials, IRS’s ACS staff could work on the PCA-type cases. Based on IRS data, ACS staffing levels—as measured by full-time equivalent (FTE) positions—have increased since fiscal year 2008. The ACS staffing levels increased by 308 FTEs from fiscal years 2008 to 2009 (i.e., 3,395 to 3,703) and were to increase by 94 FTEs during fiscal year 2010 (i.e., to 3,797). These staffing levels account for hiring done to offset attrition as well as additional hiring beyond attrition. The PDC program announcement was unclear on whether additional staff to be hired would be used for PCA-type cases. On the one hand, the announcement said that “IRS determined the work is best done by IRS employees,” which appears to refer to PCA-type work. On the other hand, the announcement also said that “new employees would give the IRS the flexibility to make assignments based on the areas of greatest need rather than filtering which cases can be worked using contractor resources.” IRS officials told us that the announcement did not imply that the collection staff to be hired in fiscal year 2009 would be used to work PCA-type cases. In comparing IRS and PCAs working the same types of cases in the PDC study, IRS was responsive to concerns of some in Congress, the National Taxpayer Advocate, and others on the study’s comparison. Although a study that adhered to federal guidance on analyzing programs would have better informed IRS’s decision on the fate of the PDC program, it is not possible to know whether such a study would have had materially different results or changed IRS’s decision on the program. However it would have provided more complete information for policymakers to consider. To the extent possible, and especially for significant program decisions, IRS should adhere to OMB and other guidance for economic analyses to better ensure that policymakers have adequate information to support their decisions. IRS does not have guidance for managers on the types of analyses that should be done and documented to support program decisions. Such analyses can yield significant benefits by helping inform decision making, but they also incur costs. Therefore, careful consideration of the potential risks and positive impacts of various study designs is necessary to select an appropriate study design and scope to answer the relevant questions in a methodologically sufficient manner. IRS’s PDC study suggested that at least certain PCA-type cases, which IRS had not been working, may be worth including in ACS’s inventory for collection action. To the extent that these results are valid and reliable, IRS may be able to make a relatively low-cost investment in certain PCA- type cases to collect tax debts. However, as we concluded our audit, IRS provided conflicting information about determining whether PCA-type cases have sufficient collection potential to be included in its collection inventory. Given the conflicting information available to us, we believe it is important that PCA-type case results are considered and incorporated as appropriate into the CDA models. If IRS determines that completing the pilot study is the best method to do so, a documented methodology and criteria for the study’s analysis could help IRS make a better decision on which PCA-type cases, if any, should be added routinely to active collection status. We recommend that the Commissioner of Internal Revenue take the following three actions: Establish guidance on the types of analyses that should be done to support decisions to initiate, renew, or expand programs. The guidance might refer to OMB Circular A-94 and, if needed, provide any supplementation specific to IRS. Establish a policy requiring documentation for the design, analyses, and conclusions of studies supporting program changes. Ensure that PCA-type case results are considered and incorporated as appropriate into the CDA model. If IRS determines completing the pilot study is the best means to ensure that PCA-type case results are considered for the CDA models, the Commissioner should ensure that the pilot study has a documented methodology and criteria to guide IRS’s analysis and decision. The IRS Deputy Commissioner for Services and Enforcement provided written comments on a draft of this report in a September 8, 2010, letter, which is reprinted in appendix I. IRS staff also provided technical comments, which we incorporated into the report as appropriate. IRS disagreed with our finding that its PDC study was not soundly designed to support its decision on whether to continue contracting out debt collection. IRS said the study’s comparison of the cost-effectiveness of PCAs and IRS working similar cases provided meaningful data that aided its decision making. IRS cited an independent review of the PDC study that found the results to be reasonable, even though the study had limitations and constraints. We continue to believe that the study was not a soundly designed cost-effectiveness comparison for supporting IRS’s decision. Our report discusses our reasoning in detail, focusing on the study’s methodological errors, narrow scope, and lack of adherence to guidance for doing such studies. For example, IRS did not do the analysis necessary to generalize the study results to the full PCA case population even though study results could differ significantly when generalized to the full population. Our meetings with staff who performed the independent review and our analyses of their documentation did not change our finding about IRS’s study. IRS agreed with our two recommendations dealing with establishing guidance on analyses to support decisions to initiate, renew, or expand a program and policies to ensure documentation of such studies. More specifically, IRS said it would review current guidance and policies and develop additional guidance where needed. IRS agreed in principle with our third draft recommendation on ensuring that a documented methodology and criteria guide IRS’s analysis and decision on whether to include selected PCA-type cases in its collection inventory, but said events have overtaken the need to complete the ongoing study, citing IRS’s plans to implement CDA models in fiscal year 2011. These models are intended to select cases with the best potential for collection action in one of IRS’s work streams. IRS said that to measure the impact of the PCA-type cases, as was the plan when the PCA project was terminated, is no longer necessary. We had discussed with IRS officials the continued need for the pilot study when IRS told us in July 2010 that it planned to implement CDA in January 2011. IRS officials, including the Acting Director, Collection Business Reengineering, said that while CDA selection would focus on collection potential and not type of case (i.e., PCA-type), the pilot study of approximately 19,000 PCA-type cases might provide data useful for improving CDA models. Officials affirmed that they initiated the pilot study because the PDC study showed that PCA-type cases might have high collection potential at low cost. Accordingly, our draft report recommended that IRS document the methodology and criteria for its pilot study. Information provided in IRS comments on the report and in response to our subsequent questions suggests that whether and how PCA- type cases may be selected for active collection inventory is uncertain. Although IRS’s comments on the draft report said that the need for completing the pilot case study was overtaken by the development of the CDA models, in separate technical comments IRS officials said they were continuing to work the pilot cases and provided no indication that they would stop working them before CDA is implemented in January 2011. Further, in response to our question about whether PCA-type case results were used in developing the CDA models, an IRS official said that they had not been used. In response to IRS’s comments and absent evidence that CDA will be implemented as planned and that its models will include IRS’s experience in attempting collection of PCA-type cases, we revised the third recommendation to better focus on ensuring that PCA-type case results are considered and incorporated as appropriate into the CDA models. Further, if IRS determines completing the pilot study is the best means to ensure that PCA-type case results are considered for the CDA models, we maintained our recommendation that IRS ensure that the study has a documented methodology and criteria to guide IRS’s analysis and decision. As agreed with your office, unless you publicly announce the contents of this report earlier, we plan no further distribution until 30 days from the report date. At that time, we will send copies to the Secretary of the Treasury, the Commissioner of Internal Revenue, and other interested parties. In addition, the report will be available at no charge on the GAO Web site at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-9110 or brostekm@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. Key contributors to this report are listed in appendix II. In addition to the contact named above, Tom Short, Assistant Director; Ray Bush; George Guttman; Lois Hanshaw; Ronald W. Jones; Veronica Mayhand; Ed Nannenhorn; Karen O’Conor; and Cynthia Saunders made key contributions to this report.

Summary: In September 2006, the Internal Revenue Service (IRS) started the private debt collection (PDC) program for using private collection agencies (PCA) to help collect some unpaid tax debts. Aware of concerns that PCAs might cost more than using IRS staff, IRS began studying the collection costs and performance of PCAs and IRS. In March 2009, IRS announced that it would not renew its PCA contracts based on the study and announced plans for increasing collection staffing. As requested, GAO is reporting on whether (1) the study was sound as primary support for IRS's PDC decision and (2) IRS has planned or made changes to its collection approach based on its PCA experience and PDC study. GAO compared IRS's study to federal and other guidance on what should be included in analyses to support program decisions and analyzed IRS's changes given expectations that IRS would consider PCAs' best practices. IRS's comparative study of the PDC program was not soundly designed to support its decision on whether to continue contracting out debt collection. Although the study was not originally intended or designed as primary support for the decision, IRS officials nonetheless used it as such. IRS did not have guidance for program managers on the type of analysis that should be done to support decisions to create, renew, or expand programs. IRS had not retained sufficient documentation on the sample used in the study or documented some analyses that would have been helpful if performed. The study results may be overstated or understated because the study sample was not generalizable to the program as a whole. The study had a narrow objective of comparing results for IRS working the same cases as PCAs had, and as a result, the study design did not consider other factors recommended by Office of Management and Budget and other guidance on conducting program analysis. For example, the study did not analyze alternatives to program scale, such as expanding it or scaling it back. Program analysis guidance states that to the extent possible all costs and benefits should be counted and alternative means of achieving a program's goals should be considered. But the study did not identify important costs and benefits, such as whether taxpayers' compliance costs would be different if IRS or PCAs work debt cases. Nevertheless, neither GAO nor IRS officials know whether the study results and decision on the program would have differed significantly if it had been designed to be primary support for IRS's PDC program. In commenting on a draft of GAO's report, IRS disagreed that the PDC study was not soundly designed. GAO stands by its analysis detailing the study's errors, narrow scope, and lack of adherence to guidance. These design and methodology deficiencies limited the study's usefulness in supporting IRS's decision. IRS has not made or planned changes to its collection approach based on its PCA experience and study. In authorizing the use of PCAs, Congress required IRS to report to Congress its measurement plan to identify any of the PCAs' best practices that IRS could adopt to improve its own collection operations. IRS did not continue to report to Congress as required. In an unpublished draft report, IRS asserted that it had reviewed a number of PCA practices and found no immediate opportunities to change its collection approach. IRS did not provide GAO documentation on the study to support that conclusion. In part because PCA-type cases had previously been considered low priority, IRS officials were surprised by the PDC study results, which indicated that IRS staff might have better results working PCA-type cases than some of the cases IRS normally works. IRS officials said that they initiated a pilot study in 2009 to help them decide whether to use IRS staff to work selected types of PCA cases. As GAO concluded its review, IRS provided conflicting information on the role of the pilot study. On one hand, IRS said a collection selection system to be implemented in January 2011 overtook the need for the study. On the other hand, an IRS official said that the results from PCA-type cases were not used in the development of the new case selection system. GAO recommends that IRS (1) establish guidance on analyses to support program decisions, (2) establish a policy requiring documentation of program studies, and (3) ensure that PCA-type case results are considered for IRS's new case selection model. IRS agreed with the first two recommendations and agreed in principle with the third, which GAO revised to reflect updated information that IRS provided.