To assess IRS' methodology, we discussed the methodology with officials in the Financial Analysis Division of the Office of the Chief Financial Officer, which was the division responsible for preparing the Compliance Initiatives Report; determined the extent to which IRS' methodology addressed the concerns we had raised in reports on prior years' compliance initiatives; discussed specific assumptions in IRS' methodology with cognizant staff in IRS' enforcement functions; assessed the sensitivity of IRS' results to changes in certain key verified the accuracy of IRS' computations. IRS computed the results of the fiscal year 1995 initiatives using data on planned and actual full-time-equivalent (FTE) staff years and planned and actual revenues for various enforcement programs. To verify the accuracy of IRS' computations, we (1) traced actual FTEs back to reports generated by IRS' Automated Financial System, (2) traced actual revenues back to reports generated by ERIS, (3) interviewed staff from IRS' enforcement functions about the methods used to develop planned FTEs and revenues, (4) recomputed calculations, and (5) resolved any inconsistencies with cognizant IRS staff. We did not assess the reliability of the reports generated by ERIS or the Automated Financial System. We did our work from June 1996 to May 1997 in accordance with generally accepted government auditing standards. We requested comments on a draft of this report from the Acting Commissioner of Internal Revenue or his designated representative. The Acting Commissioner provided comments in a letter dated July 24, 1997. Those comments are reprinted in appendix II and are summarized and evaluated at the end of this letter. In reviews of past compliance initiatives, we identified several weaknesses in IRS' methodology for computing and tracking initiative results. Our past reviews disclosed, for example, that IRS had overstated initiative results by failing to recognize the opportunity costs associated with moving experienced staff off-line to train new staff and by failing to adequately account for underrealizations of planned base staffing, was unable to track actual enforcement results, and was not providing Congress with enough meaningful information on initiative results because it was reporting positive results from initiatives without recognizing negative results from reductions in base activities. Over the years, IRS revised its methodology to address those, and other, concerns. In preparing its fiscal year 1995 Compliance Initiatives Report, for example, IRS recognized the impact of opportunity costs, obtained revenue data from an automated system (ERIS) that was designed to track actual enforcement results, and improved the report's usefulness to Congress by including not only the estimated results of the initiatives but also the estimated results of the base enforcement programs and explanations for variances between the results anticipated when the initiatives were approved and the estimated final results. Also, in computing the results of the fiscal year 1995 compliance initiatives, IRS adopted a rule that no FTEs, and thus no revenue, would be allocated to the initiatives until planned base staffing had been achieved. Although the methodology used for the fiscal year 1995 initiatives is an improvement over previous methodologies, the results of that methodology are estimates that are sensitive to various productivity assumptions. Those productivity assumptions were not based on empirical data and could, if they were erroneous, cause IRS' reported results to be overstated or understated. Also, in verifying IRS' calculations, we found four errors that had a relatively minor effect on IRS' reported results. IRS developed and uses ERIS to track the results of enforcement activities. Assuming ERIS does what it is designed to do, it should provide the total amount of dollars actually collected as a result of enforcement activities in fiscal year 1995 (i.e., $31.4 billion). However, as IRS acknowledged in the Compliance Initiatives Report, ERIS does not distinguish between the dollars collected from base enforcement activities and the dollars collected as a result of the initiatives. Similarly, the Automated Financial System, from which IRS extracted the total number of FTEs spent on enforcement activities in fiscal year 1995, does not distinguish between staffing for base activities and for the initiatives. Because its systems do not distinguish between base and initiative activities, IRS, as part of its methodology, developed the formula we describe in appendix I to allocate the $31.4 billion in enforcement revenue between base and initiative activities. Before implementing its new methodology, IRS briefed us on the allocation formula. We said then that because IRS had no initiative-specific data, its formula was a reasonable approach for identifying initiative results. We continue to believe that. However, because planned (i.e., estimated) revenue and staffing levels are an integral part of that formula, the end results are estimates. Thus the "actual" initiative results cited in the Compliance Initiatives Report are not actual results but are estimates of results. IRS did not clearly disclose that fact in the Compliance Initiatives Report. In tables throughout the report, for example, IRS refers to "actual," without clearly explaining the term. One important set of assumptions embedded in IRS' methodology relates to the comparative productivity of new staff versus experienced employees. IRS used these assumptions in computing planned revenue, which was an integral part of the methodology. The two primary IRS enforcement functions, Collection and Examination, approached the issue of relative productivity differently. Collection assumed that new staff were less productive than experienced staff, even after they were trained, while Examination believed that new staff, once trained, were as productive as experienced employees. Neither of these assumptions was based on empirical data. According to IRS officials, to estimate the relative productivity of new Collection staff, Collection Division officials met in a brainstorming session and decided, based primarily on their institutional knowledge, that new staff were generally 50 percent as productive as experienced employees during their first year on the job. Part of that reduced productivity assumed by Collection is attributable to the amount of time new staff spend in training and part to the belief that it takes time for a new employee to become as productive as an experienced one. (Collection assumed that new employees do not reach full productivity until their second or third year, depending on their position). We have no basis to determine whether Collection's productivity assumptions are correct. We do know, however, that changes to the assumptions could significantly alter the reported results of the compliance initiatives. To demonstrate the sensitivity of the reported results to changes in Collection's productivity assumptions, we arbitrarily adjusted Collection's 50-percent assumption by 5 percentage points in either direction and recalculated the initiatives' results. Our recalculation showed that a 5 percentage point change would either increase or decrease IRS' reported initiative results of $545.2 million by $42 million (about 8 percent), depending on the direction of the change. Unlike Collection, the Examination function assumes that new staff are as productive as experienced staff after they have completed classroom training. That assumption applies to all of Examination's enforcement staff--tax examiners, who audit simple issues by corresponding with taxpayers; tax auditors, who do more complex audits generally by meeting with taxpayers at an IRS office; and revenue agents, who do the most complex audits generally by meeting taxpayers or their representatives at the taxpayer's home or place of business. Examination officials told us that they did not have any empirical data to support their assumption that new tax examiners are as productive as experienced tax examiners because Examination's information systems do not track individual tax examiner's accomplishments. Thus, historical data cannot be separated between experienced and inexperienced tax examiners. Instead, Examination officials justified their assumption by noting that tax examiners work on relatively noncomplex issues and returns and are able to get up to speed fairly quickly. However, according to an IRS official, after 2 weeks of formal classroom training, tax examiners are assigned to an on-the-job instructor for 10 weeks. The need for on-the-job instructors suggests that IRS does not expect new employees to be able to handle issues and returns as effectively or efficiently, and thus be as productive, as experienced employees. Examination officials also said that they did not believe that it was necessary to assume different productivity levels for experienced revenue agents and tax auditors and new agents and auditors. The officials provided two reasons for their position. First, Examination adjusts its revenue estimates to consider the amount of time that new staff must spend in classroom training. Examination assumes, based on past experience, that the amount of time available to do audit work is reduced between 19 and 25 percent (it varies between revenue agents and tax auditors) because of classroom training requirements. However, that adjustment only affects the amount of time new staff have to do audits; it does not get at the issue of how productive new staff are when they are doing audits. Second, Examination's resource allocation model imputes a lower marginal yield for each additional return audited by initiative revenue agents and tax auditors. That means, in effect, that Examination assumes that each additional return audited will generate less revenue than the return audited before it. However, although IRS assumes a lower yield with each additional audited return, it does not consider any differences in the efficiency with which experienced staff and new staff audit tax returns. In that regard, the officials said that, after completion of classroom training, new staff are expected to close the same number of cases in any year as experienced employees who work the same types of cases. But the productivity of Examination staff is determined not only by the number of cases they close in a year but also by the revenue generated from those cases. Thus, even if new staff were to close as many cases as experienced staff (Examination officials could provide no evidence to support that contention), they may or may not achieve comparable dollar results. According to an Examination official, revenue agents and tax auditors, during their first year on the job, have 10 weeks and 13 weeks, respectively, of on-the-job training after their classroom training. The need for on-the-job training suggests, in our opinion, that new staff may not be prepared to be as productive as experienced staff, after classroom training. Although we did not have data to test the appropriateness of Examination's productivity assumption, we tested the sensitivity of IRS' reported initiative results to changes in that assumption by assuming the following: New tax auditors would be 75 percent as productive as experienced tax auditors. New revenue agents would be 69 percent as productive as experienced revenue agents. New tax examiners would be 95 percent as productive as experienced tax examiners. We arrived at the 75-percent and 69-percent figures for tax auditors and revenue agents, respectively, by using Collection's productivity assumption of 50 percent and adjusting it to recognize the fact that Examination's assumption already includes a factor for lost productivity due to training. We assumed only a slight fall-off in productivity for tax examiners because the correspondence audits they do are straightforward, and thus, the skills needed to do them can be more quickly learned than the skills needed by tax auditors and revenue agents. Use of the different productivity assumptions reduced the initiative results by $12.9 million--a reduction of about 5 percent from the reported amount of $237.1 million that IRS estimated was generated by Examination through the initiatives. We traced the data in the Compliance Initiatives Report back to supporting documentation and verified the various calculations involved in using IRS' methodology. We found four relatively minor errors. The first two errors involved the categorization of FTEs between revenue- and nonrevenue-producing FTEs. A portion of the staffing increase associated with compliance initiatives is for support staff, such as clerks and secretaries. Those staff, unlike revenue officers, revenue agents, and other frontline staff, do not directly generate revenue. Thus IRS, in its calculations, segregated nonrevenue-producing FTEs from revenue-producing FTEs. In verifying that part of IRS' calculations, we found two errors. IRS mistakenly (1) categorized about 140 nonrevenue-producing FTEs in the Collection function as revenue-producing and (2) included 240 nonrevenue-producing Examination FTEs in the formula it used to allocate enforcement revenue between base activities and initiatives. The third error involved the use of an incorrect average yield figure in computing initiative results for the Compliance Research program. The fourth error involved a failure to include about 180 tax examiners in computing the initiative results for the Collection function. By our calculations, these four errors--which IRS officials acknowledged--caused IRS' reported yield from the initiatives to be understated by $2.6 million. Absent other changes, correction of the four errors would increase IRS' reported yield to $805.9 million. In considering IRS' estimates of the results of the fiscal year 1995 compliance initiatives, there are two other caveats that are relevant: (1) the fact that IRS collected a certain amount in fiscal year 1995 as a result of the initiatives does not necessarily mean that IRS collected more enforcement revenue in fiscal year 1995 than in fiscal year 1994 but only that IRS collected more in fiscal year 1995 than it estimated it would have without the initiatives and (2) the first year's results from the fiscal year 1995 initiatives are not necessarily indicative of what other compliance initiatives would generate in their first year. Although IRS reported that the compliance initiatives resulted in additional collections in fiscal year 1995 of $803.3 million, it does not mean that IRS brought in $803.3 million more than it did in fiscal year 1994. What it means is that IRS estimated that it generated $803.3 million more in enforcement revenue in fiscal year 1995 than it had estimated it would generate without the compliance initiatives. According to the Compliance Initiatives Report, IRS collected a total of $31.4 billion in fiscal year 1995--$30.6 billion from its base programs and $0.8 billion from the initiatives. Despite the estimated additional revenue from the initiatives, however, the amount of enforcement revenue collected in fiscal year 1995 was less than the amount collected in fiscal year 1994. That is, IRS data indicated that IRS collected about $33.1 billion in fiscal year 1994, or $1.7 billion more than in fiscal year 1995. According to IRS officials, a number of factors could make enforcement revenue decrease even with a staffing increase in the same year. Revenue collected in one fiscal year is a function not only of that year's staffing but also of prior years' staffing. Much of the revenue impact of a staffing increase or decrease occurs in subsequent years because of the possibility of appeals, litigation, and collection activity. In that regard, IRS officials said that enforcement revenue in fiscal year 1996 increased to $38.0 billion, even with a staffing decrease, partially as a result of the fiscal year 1995 compliance initiatives. A second caveat to keep in mind is that IRS' results in fiscal year 1995 are not necessarily indicative of the results that would be achieved in the first year of future compliance initiatives. The results achieved for any compliance initiative depend on many factors that can, and most likely will, vary from one initiative to another. Two of those factors, both of which had a significant impact on the results achieved in fiscal year 1995, are (1) the extent to which new staff must be hired to fill the positions authorized by the initiatives and (2) how IRS decides to allocate the initiative positions among its various enforcement programs. Most of the positions funded with the $405 million provided for the compliance initiatives in fiscal year 1995 were filled not by new hires but by staff who were already on board. This happened, at least in part, because IRS' fiscal year 1995 appropriation, except for the compliance initiatives, actually resulted in reductions in IRS' enforcement staffing. Thus, many of the positions funded by the $405 million were used to offset that reduction. According to data provided by IRS, of the 5,470 initiative positions filled as of September 30, 1995, 1,145 were filled by new staff. The other 4,325 were filled by existing employees either through lateral reassignments or promotions (such as promoting tax auditors to fill revenue agent positions). If future initiatives require a greater proportion of new staff, the results could be different from those in fiscal year 1995 because new staff (1) require more training, which, under IRS' current procedures for training new staff, increases the opportunity cost associated with moving experienced staff off-line to do the training and (2) generally can be expected to generate less revenue, at first, than experienced staff. Decisions on how to allocate staff among enforcement programs also affects initiative results. The change in IRS' estimate of how much revenue would be generated by the fiscal year 1995 initiatives, which we noted at the beginning of this report, is an example of how staffing decisions can affect initiative results. IRS' first estimate was $9.2 billion over 5 years and $331 million in fiscal year 1995. Then, in an effort to maximize revenues, IRS decided to allocate more of the $405 million to areas, such as Automated Collection System sites, that are staffed by lower graded personnel and to allocate fewer dollars to more costly areas, such as the Collection Field Function, which is staffed by higher graded revenue officers. This reallocation enabled IRS to fund many more FTEs than originally expected and resulted in revised estimates of $9.6 billion over 5 years and $728 million in fiscal year 1995. IRS' methodology for computing the results of the fiscal year 1995 compliance initiatives is a significant improvement over past methodologies. However, there are productivity assumptions embedded in the methodology that are not based on empirical data and could cause the results of an initiative to be overstated or understated. We do not have a basis for determining what the correct assumptions should be, but our sensitivity analyses showed that a change in the assumptions used could have a significant effect on the reported initiative results of $803.3 million. For example, as discussed earlier, changing Collection's productivity assumption by 5 percentage points would either increase or decrease the reported results by $42 million and changing Examination's assumptions to be comparable to Collection's would decrease the reported results by $12.9 million. After adjusting IRS' reported results for the effect of these different productivity assumptions and after increasing the results to account for the $2.6 million in calculation errors, the estimated yield from the fiscal year 1995 compliance initiatives would fall somewhere between $751.0 million and $847.9 million. We requested comments on a draft of this report from the Acting Commissioner of Internal Revenue. IRS provided comments in a letter dated July 24, 1997 (see app. II). Overall, IRS agreed with the findings in the report, which it said confirmed that IRS accurately tracked and reported on the results of the compliance initiatives. However, IRS expressed concerns about two aspects of the observations in the report. First, IRS took issue with our statement that it did not clearly disclose the fact that the initiative results cited in the Compliance Initiatives Report were not "actual" results but rather estimates. To show that it had clearly disclosed that fact, IRS quoted two passages from the Compliance Initiatives Report which refer to the allocation of the revenue to base and initiative activities. In our opinion, the statements IRS quoted, while informative to a careful reader, do not provide sufficient information to make clear that the results are estimates, especially when every table in the final report referred to the results as "actuals." Second, IRS observed that, although we state that we did not assess the reliability of the data in ERIS, (1) after reviewing the Compliance Initiatives Report and supporting ERIS data, we have found no indication that ERIS does not do what it purports to do--accurately accumulate and summarize enforcement revenue; (2) work done to date on ERIS as part of our financial audit of IRS had not disclosed any problems; and (3) an ERIS sample we took as part of an audit of large corporations concluded that ERIS does what it purports to do. IRS also observed that we have asked for ERIS data in conjunction with two recently initiated audits. We appreciate IRS' viewpoint; however, (1) the scope of our review of the Compliance Initiatives Report, as discussed earlier, did not include a specific assessment of ERIS reliability; (2) we have not yet done sufficient work on ERIS as part of the financial audit to reach any overall conclusion about data reliability; and (3) the scope of our audit of large corporations was also not broad enough to reach any overall conclusion about ERIS reliability. Until such time as ERIS' reliability can be determined, we necessarily continue to rely on ERIS data for some of our work because they are the best data available. However, the standards applicable to our work require that we disclose that data reliability has not been confirmed. We are sending copies of this report to the Committee Chairman; the Chairmen and Ranking Minority Members of the Senate Committee on Finance, the House Committee on Ways and Means, and the House Committee on Government Reform and Oversight; various other congressional committees; the Secretary of the Treasury; the Commissioner of Internal Revenue; the Director of the Office of Management and Budget; and other interested parties. We will also make copies available to others upon request. Major contributors to this report are listed in appendix III. If you have any questions, please contact me on (202) 512-9110. According to the Compliance Initiatives Report, IRS estimated the amount of revenue attributable to the fiscal year 1995 compliance initiatives by means of a general formula that consists of two components. The first component is the ratio of total actual yield per FTE (base and initiative combined) divided by the total planned yield per FTE (base and initiative combined). This component indicates by how much the average tax yield was over or under what IRS expected. If IRS accurately predicted the average yield, the ratio value of the first component would equal 1. If the actual yield was more or less than what was predicted, the ratio value would be greater or less than 1. The second component multiplies the number of "actual" FTEs allocated to the initiatives by the planned yield per initiative FTE. The result of the second component is then multiplied by the result of the first component. If the first component equaled 1, the yield attributable to the initiatives would be equal to the number of initiative FTEs times what yield they were predicted, on average, to realize. However, if the first component was less than 1, implying that the average yield of both base and initiative staff was less than expected by some percentage, the expected tax yield per initiative FTE would be reduced by the same percentage. Conversely, if the ratio value of the first component was greater than 1, the second component would be increased by the percentage by which the actual average yield exceeded what was expected. To illustrate, assume the following information: Total actual yield -- $70 million Actual number of FTEs (base and initiative) -- 1,800 Actual yield per FTE ($70 million divided by 1,800 FTEs) -- $38,889 Planned total yield -- $85 million Planned number of FTEs (base and initiative) -- 1,500 Planned yield per FTE ($85 million divided by 1,500 FTEs) -- $56,667 First component: $38,889 divided by $56,667 = 0.686 or 68.6 percent. Planned initiative yield -- $5 million Planned initiative FTEs -- 400 Planned yield per initiative FTE -- $12,500 "Actual" initiative FTEs -- 700 Second component: 700 times $12,500 = $8,750,000. "Actual" initiative yield: 0.686 times $8,750,000 = $6,002,500. The above information is an illustration. The numbers and thus the results would vary among initiatives and even within an initiative. For example, Examination assigned its initiative staff to various audit classes ("individual taxpayers who file a Form 1040C showing total gross receipts of less than $25,000 dollars" is an example of an audit class). For each audit class, IRS would apply the above formula; and for each audit class, the numbers and the results would be different. David J. Attianese, Assistant Director James A. Wozny, Assistant Director John Lesser, Evaluator-in-Charge Charles C. Tuck, Economist The first copy of each GAO report and testimony is free. Additional copies are $2 each. Orders should be sent to the following address, accompanied by a check or money order made out to the Superintendent of Documents, when necessary. VISA and MasterCard credit cards are accepted, also. Orders for 100 or more copies to be mailed to a single address are discounted 25 percent. U.S. General Accounting Office P.O. Box 37050 Washington, DC 20013 Room 1100 700 4th St. NW (corner of 4th and G Sts. NW) U.S. General Accounting Office Washington, DC Orders may also be placed by calling (202) 512-6000 or by using fax number (202) 512-6061, or TDD (202) 512-2537. Each day, GAO issues a list of newly available reports and testimony. To receive facsimile copies of the daily list or any list from the past 30 days, please call (202) 512-6000 using a touchtone phone. A recorded menu will provide information on how to obtain these lists.

Summary:
Pursuant to a congressional request, GAO reviewed the Internal Revenue Service's (IRS) Fiscal Year (FY) 1995 Compliance Initiatives Report, focusing on: (1) the methodology IRS used to allocate staff years and revenues between the base enforcement programs and the compliance initiatives; and (2) certain caveats to consider in interpreting IRS' reported results. GAO noted that: (1) IRS could not compile actual revenue results from the FY 1995 compliance initiatives because the Enforcement Revenue Information System (ERIS) only provides information on the total amount of revenue collected as a result of enforcement activities and because other systems, such as those that track IRS staffing, also do not account separately for base enforcement activities and initiative activities; (2) therefore, IRS developed a new methodology to allocate FY 1995 enforcement revenues between base programs and the initiatives; (3) IRS' new methodology: (a) accounted for the opportunity costs associated with moving experienced staff off-line to train new staff; (b) provided that no staff or revenue would be allocated to the initiatives until planned staffing for base programs had been achieved; and (c) improved the Compliance Initiatives Report's usefulness to Congress by including total staffing and total revenue for the various enforcement programs, allocated between base and initiatives, along with explanations for variances between the results anticipated when the initiatives were approved and the estimated final results; (4) although the methodology used for the FY 1995 initiatives is an improvement over previous methodologies, the results of that methodology are estimates that are sensitive to assumptions embedded in the methodology about the productivity of new staff and more experienced staff; (5) those assumptions were based on the judgments of IRS managers rather than empirical data; (6) GAO does not know what the correct assumptions are, but GAO's sensitivity analyses showed that a change in productivity rates could have a significant effect on the reported results; (7) in considering IRS' estimates of the FY 1995 compliance initiatives, there are two other caveats that are relevant; (8) the fact that the initiatives generated a certain amount of revenue in FY 1995 does not necessarily mean that IRS collected more enforcement revenue in FY 1995 than it did in FY 1994 but only that IRS collected more enforcement revenue in FY 1995 than it had estimated it would collect without the initiatives; (9) in fact, the amount of enforcement revenue IRS reported collecting in FY 1995 was less than that reported for FY 1994; and (10) because, the estimates of revenue attributable to the compliance initiatives depended on various assumptions, including how IRS decided to allocate staff, the results in FY 1995 are not necessarily indicative of what other compliance initiatives would generate in their first year.