IRS Form 990-series return or notice must be filed by most organizations exempt from income tax under Internal Revenue Code section 501(a), and certain political organizations and nonexempt charitable trusts. TE/GE uses Form 990 reporting for promoting compliance and enforcing federal tax law for tax-exempt organizations (see appendix II for a copy of the Form 990 and a list of its schedules). Form 990 asks for information about an organization such as: employees, governance, and compensation; revenue and expenses; assets and liabilities; employment tax compliance; and specific organizational issues, such as lobbying by charities and private foundations. TE/GE redesigned the Form 990 for the first time in nearly 30 years for tax year 2008, and has made subsequent changes to the form (see appendix III for a summary of the changes). For tax year 2017, which is the most recent year of completed filing data, organizations filed 319,183 Form 990s. Beyond the basic Form 990, other versions include: Form 990–EZ, Short Form Return of Organization Exempt from Income Tax. This form reduces the filing burden on small tax-exempt organizations. Organizations with less than $200,000 in gross receipts and less than $500,000 in total assets may use it. For tax year 2017, 232,764 Form 990-EZ’s were filed. Form 990–N, Electronic Notice (e-Postcard) for Tax-Exempt Organizations Not Required to File Forms 990 or 990–EZ. Most small organizations whose annual gross receipts are normally $50,000 or less may file Form 990-N. For tax year 2017, 652,280 Form 990-N’s were filed. Form 990–PF, Return of Private Foundation. In addition to private foundations, nonexempt charitable trusts treated as private foundations are required to file Form 990-PF. For tax year 2017, 113,658 Form 990-PF’s were filed. Certain larger organizations are required to electronically file their returns. The Taxpayer First Act of 2019 requires all organizations to electronically file Form 990’s for tax years beginning after July 1, 2021. TE/GE can assess financial penalties for failing to file a required Form 990. As an employer, or if an exempt organization generates unrelated business income, additional tax reporting requirements may apply, such as for employment tax or unrelated business income. A 2017 TE/GE reorganization created CP&C to provide a centralized approach to compliance planning, examination selection and assignment and planning and monitoring activities. CP&C has three groups as follows: 1. Issue Identification and Special Review identifies and develops issues for examinations or compliance activities and certain criteria for examination selection. 2. Classification and Case Assignment uses IRS staff known as “classifiers” to review returns for examination under different examination sources (see appendix IV). Classification is the process of determining whether a return should be selected for compliance activities, what issues should be the primary focus of the compliance activity, and the type of compliance activity that should be conducted. 3. Planning and Monitoring develops an annual work plan and monitors performance. The work plan details the number of examination starts, closures and other measures. It develops classification requests to ensure that enough returns are available to meet work plan goals. TE/GE’s Compliance Governance Board (Governance Board) oversees TE/GE’s compliance program, including CP&C operations such as approving priority issue areas—known as compliance strategies. The Governance Board also reviews program goals, considers metrics and reporting, and reviews the performance of compliance strategies. The Governance Board has five TE/GE executives plus counsel who are voting members as well as three non-voting members. The Exempt Organizations examinations group is responsible for compliance activities. Examinations have various outcomes for an organization. The most severe outcome is revocation of tax exempt status. Taxes—such as employment or excise—may be assessed as a result of an examination. In fiscal year 2019, approximately $131 million in taxes were assessed. TE/GE conducts compliance contacts—non-examination correspondence such as compliance checks and soft letters—that are used to handle some compliance issues. For example, compliance checks determine whether specific reporting or filing requirements have been met. A “soft letter” notifies an organization of changes in tax-exempt law or potential compliance issues. A response to these letters is not required. TE/GE also reviews tax-exempt hospitals for compliance with certain community benefit requirements. In fiscal year 2019, TE/GE closed 1,470 compliance checks, sent 3,955 soft letters, and closed 750 hospital reviews. Compliance checks and hospital reviews can result in an examination while responses to soft letters may result in a compliance check. TE/GE identifies exempt organization returns for examination from many sources and categorizes examinations into three groups, known as portfolios: (1) Data Driven Approaches, (2) Referrals and Other Casework, and (3) Compliance Strategies. All three rely on data, to some extent, to make decisions on selecting returns for examination. This portfolio uses analytical models and queries based on quantitative criteria to identify potential examinations. TE/GE has three separate models that review exempt organization data from Forms 990, 990-EZ, and 990-PF for compliance. The models “score” returns for examination based on potential noncompliance. The Form 990, 990-EZ and 990-PF models have 354 unique queries. For purposes of this report, a query reviews databases to identify responses on returns that may indicate noncompliance because they do not meet certain criteria or expected values, such as exceeding a dollar threshold. Exempt Organizations Examination staff developed many of the queries, based on information collected on the Form 990 after it was redesigned for tax year 2008, according to TE/GE officials. As queries were developed, staff tested and used them to identify certain potentially noncompliant populations and to identify returns that were flagged by multiple queries. Starting in fiscal year 2016, TE/GE began using queries in models. The models use a scoring system that applies weights, or points, to each query result to generate a score—which for the Form 990 model has ranged from zero to more than 50—for a return. The models also screen out returns that are approaching a statute of limitations date, if the organization is not active, or has a current or recent examination history. Since November 2017, staff have been able to submit potential compliance issues for consideration through an online submission portal for Governance Board approval. CP&C has the option of considering whether these ideas result in model changes, according to IRS officials. Twice a year, each model is run using the latest data, and generates a Model Score Sheet (MSS). The MSS is a ranked list of returns that score above a minimum threshold. A classifier uses the ranking to identify returns for potential examination. Although the models screen for examination status and statute of limitations, a TE/GE official said the classifier also checks whether the statute of limitations date is near and whether the organization recently had undergone an examination or compliance check, as well as whether the return was identified under another selection method. This official explained that a classifier checks the criteria because conditions may have changed since the model’s last run. The classifier selects returns to fulfill a stocking plan, which identifies the number and type of returns to be examined to meet work plan requirements. See figure 1. Aside from the three models, TE/GE also uses other methods and data to identify and develop compliance work. The Data Driven Approaches portfolio includes approaches that TE/GE developed in partnership with IRS’s RAAS division. The partnership began in 2016 and continues today, according to IRS officials. The portfolio also includes some of the queries that TE/GE ran prior to fiscal year 2016 for examination selection. Some of these examinations remained open as of fiscal year 2019. Although not all of the returns selected for examination in this portfolio rely on data for examination selection, we describe them all below. Referrals. Referrals are complaints about exempt organization noncompliance made by third parties, including the public and other IRS offices or divisions. Post Determination Compliance. Sampling and queries are used to identify organizations that file Form 1023-EZ. Claims. Claims are requests for tax refunds, adjustments of tax paid, or credits not previously reported or allowed. Form 990 Queries (pre-model): These queries were run prior to fiscal year 2016. Some of these examinations remained open as of fiscal year 2019. Training. TE/GE uses these examinations, selected based on various methods, to teach examiners. Other Projects. TE/GE initiated these examinations under older compliance projects, using a variety of selection methods. The Compliance Strategies portfolio consists of compliance issues that originated from a Compliance Issue Submission Portal for TE/GE staff. The strategies are approved by the Governance Board, which results in adding the compliance strategy to the work plan. In fiscal year 2019, TE/GE closed examinations under three compliance strategies, including private foundation loans, and for-profit entities that converted to 501(c)(3) organizations. Returns are selected using sampling or other uses of data. Table 1 shows examinations closed for the three portfolios. Once an examination is underway, an examiner may expand it to include an organization’s returns for other tax years or other types of returns such as employment tax returns. IRS refers to these additional examinations as “pick-ups,” each of which is counted as a separate examination. Examiners must obtain manager approval to expand an examination. Examiners are required to check that an organization filed all returns that are required. If the examiner finds that a return was not filed—such as an employment tax return—and is unable to secure the return, he or she may prepare a “dummy” return called a substitute for return (SFR). The organization’s activities, records, and documents may then be examined. In 2017, TE/GE hired a contractor to assess aspects of the exempt organization process for examination selection, with a focus on the Form 990 model. In January 2018, the contractor released a report on the development and operation of the models. The contractor released a second report in July 2018 on the Form 990 model performance. The contractor found the model was not always identifying the “next best case” as TE/GE intended because scores did not consistently predict certain measures of noncompliance. Across both reports, the contractor made 17 recommendations, which we discuss later in this report (see appendix V). As of March 2020, TE/GE implemented one recommendation on model update submissions and part of another on hiring assessments. In September 2019, TE/GE initiated another study with the same contractor—with a planned release of the report in September 2020—on developing alternatives to the Form 990 model. Since the Form 990 model was first run for fiscal year 2016, the percentage of examinations closed that were identified by using data, such as through models or queries, has increased each year, as shown in figure 2. Almost half of these examinations are from the models. This increased reliance on using data in selecting returns for examination offers potential efficiencies. For example, a potential efficiency from using data to find possible noncompliance could mean fewer steps for staff who classify returns. Ultimately, this could allow TE/GE to shift staff from classifying returns to doing compliance activities such as examinations to confirm any actual noncompliance. Another potential efficiency would be selecting more examinations that find changes to the return. To measure the outcomes of examinations, TE/GE computes a “change rate,” or the percentage of closed examinations with a change to the return. In general, a higher change rate indicates that more examinations found noncompliance. Examinations selected using data have a slightly better change rate than other selection sources (84 percent versus 82 percent) for closures in fiscal years 2016 through 2019. Similar to all examinations that used data, the change rate for examinations selected using data through the Form 990 model (87 percent) was higher than the change rate for other selection sources (82 percent) in fiscal years 2016 through 2019. However, we found evidence that the changes identified in examinations did not clearly result from using the Form 990 model’s scoring system. Specifically: The model has not improved change rates compared to pre-model Form 990 queries. A higher model score is not associated with a higher change rate. Most examination changes credited to the model come from pick-up returns and SFR’s that examiners identify rather than from primary returns identified by the model score. The scoring generated by the Form 990 model has not improved change rates compared with the Form 990 queries that TE/GE used prior to the model. The change rates for both the Form 990 model and the pre-model queries, for fiscal years 2016 through 2019, was 87 percent. Similarly, for the last 2 fiscal years, the change rate for all Form 990 models was roughly equivalent to the change rate for other selection sources of exempt organization examinations. As shown in table 2, the models had a slightly higher change rate in fiscal year 2018, and a slightly lower change rate in 2019, compared to the other sources. Form 990 model scores for returns do not consistently predict examination change rates based on our analysis of examination closures since the model’s first run in 2016 through fiscal year 2019; the scores better predicted the rate at which returns were selected for examination. See figure 3. The figure shows little relationship between model scores and change rates; change rates remained relatively flat as model scores increase. While change rates were slightly higher for the less than 1 percent of returns scoring 45 or above relative to lower-scoring returns, TE/GE only examined 65 returns during fiscal years 2016 through 2019 that scored this high. The overall correlation between model scores and change rates is -.02. A TE/GE official said that it is not difficult to find a small issue on a return, which allows for a change regardless of score. To attempt to measure the severity of an examination change, TE/GE developed a weighted disposal score (WDS). However, TE/GE does not have documented criteria or justifications for how the weights were developed. A TE/GE official acknowledged that TE/GE has not used WDS because of questions about how consistently the weights have been developed. If WDS was to be used as a measure, TE/GE would need to ensure the adequacy of the support for the related weights and scores. According to TE/GE’s fiscal year 2020 Program Letter, the model relies on quantitative criteria, “which allows TE/GE to allocate resources that focus on issues that have the greatest impact.” To the extent that a higher model score does not predict a higher change rate, the model is not selecting returns with the greatest impact. Further, taxes assessed per return also indicate that examinations are not having the greatest impact. For fiscal years 2016 through 2019, the examinations credited to the model averaged $2,460 in proposed tax assessments per return, compared with an average of $19,042 for the rest of the exempt organization examinations. TE/GE acknowledged that its scoring methods are limited because it does not utilize modern data practices. It contracted for a study, to be completed in September 2020, of alternative model architectures and scoring methods that incorporate best practices for using criteria and options for scoring returns. As shown in table 3, the Form 990 model scoring did not account for most closed examinations and examination changes credited to the model during fiscal years 2016 through 2019. Rather, examinations of “pick-up” returns and substitutes for returns (SFRs) accounted for most closed examinations and produced a higher change rate than examinations of primary returns scored by the model. Examiners find these other returns during examinations of returns identified by the model. The higher change rates for pick-up and SFR returns compared to the primary returns identified by the model support TE/GE’s policy to examine all pick-up returns and SFRs that meet examination criteria. However, this raises questions about how well the model identifies noncompliant returns. Given the lower change rate for the returns the model scored, the queries for noncompliance on the Form 990 may not be effective. While the model includes queries on noncompliance related to “pick-up” issues such as unfiled employment tax returns, the necessary data were not available to allow us to analyze how often these queries identified the primary return for potential noncompliance. As discussed later, an analysis of queries could provide insight into the validity of the model. Internal control should be an integral part of an agency’s operational processes and structure to help managers achieve their objectives on an ongoing basis. When evaluating implementation, management determines if the control exists and is operational. A deficiency in implementation exists when no such control is present or is not implemented correctly, preventing objectives from being met. Documentation is required to show the effective design, implementation, and operation of an internal control system. The level and nature of documentation can vary based on the size of the agency and the complexity of its processes. Management exercises judgment in determining the extent of documentation that is needed. TE/GE has not fully implemented or documented internal controls for analyzing data for examination selection, meaning it cannot be assured that its selection decisions will produce the desired outcomes. The internal controls range from two controls that TE/GE adequately documented and implemented to seven others where TE/GE did not. The seven include five controls presented as sequential steps in using data for making selection decisions as well as two controls addressing timely documentation of Internal Revenue Manual (IRM) sections and risk management. The first internal control TE/GE implemented involved assessing staff competence. To ensure competence in using data to make decisions, TE/GE officials contracted with data specialists for modeling expertise to incorporate statistical and machine learning into examination selection. Bringing in this modeling expertise was an important step because exempt organization examinations staff, rather than statisticians or data analysts, initially developed the examination selection models, according to TE/GE officials. TE/GE also provided documents on training and basic duties for staff when analyzing data. What are Internal Controls and Why Do They Matter? One way federal agencies can improve accountability in achieving their missions is to implement an effective internal control system. Effective internal control comprises the plans, methods, policies, and procedures used to fulfill objectives on an ongoing basis. It serves as the first line of defense in safeguarding assets and increases the likelihood that an agency will achieve its objectives while adapting to changing environments, demands, risks, and priorities. Effective internal control provides reasonable, not absolute, assurance that an organization will meet its objectives. The second internal control involved communicating inside and outside of TE/GE. Internally, TE/GE staff could provide feedback through an online compliance issue submission portal in fiscal year 2018. Submissions may become compliance strategies or model queries. As for external communication, TE/GE collaborated on data-related issues in an IRS- wide group and with statistical specialists in the RAAS division. For example, RAAS identified potential data sources for compliance issues and drew samples for certain compliance strategies to test rates of noncompliance. In addition, to show how it communicates essential information with staff and outside parties, TE/GE provided examples on disseminating guidance and examination accomplishments, including examination starts and closures. TE/GE did not fully implement and document internal control over the processes and data used to select returns for examination. These processes cover five key steps for using data to decide which returns to select for examination (see figure 4). Effective internal controls would enable TE/GE to show how feedback and lessons learned in Step 5 can help it better determine how to create and use quality information (Step 3) and what decisions to make (Step 4) when pursuing the established objectives (Step 1). However, TE/GE has not defined measurable objectives or undertaken regular evaluations to assess progress toward objectives. Although TE/GE was able to describe its approach for accessing relevant and reliable data, processing those data into quality information and using the data to make decisions, it was not able to fully document how its control processes worked, as discussed below. Since its 2017 reorganization, TE/GE has not established measurable objectives to select exempt organization returns for examination (see figure 5). Specifically, TE/GE has not produced formal objectives that are aligned with its mission and the IRS strategic plan, are expressed in quantitative terms, and are related to examination selection and program outcomes. TE/GE documents, including Program Letters and Business Performance Reviews, refer to outcomes that could constitute objectives—such as improving the models and advancing data analytics to drive decisions about identifying and addressing existing and emerging areas of noncompliance—but they do not identify them as such. TE/GE officials acknowledged the need to establish measurable objectives. They said their efforts are evolving and they need to improve analytical abilities to help assess the capacity for meeting objectives. For example, one official said they are working to establish objectives at the onset of a compliance strategy. Without measurable and defined objectives, TE/GE cannot effectively analyze how well it selects returns for examination and lacks a clear vision of what it is trying to achieve. A lack of measurable objectives also hinders implementing other internal controls, such as evaluating performance or assessing risk, as discussed later. The IRM has procedures for processing Form 990 data, which include controls over acceptance and transmission of the data (see figure 6). TE/GE provided data that showed error rates for electronically filed returns filed in 2019 were between 1 and 4 percent. However, taxpayer or transcription error rates for paper returns filed in 2019 were between 19 and 32 percent of filed returns, depending on the version of the Form 990. TE/GE was not able to show that it regularly reviews and remediates such errors to ensure the reliability of Forms 990 data. However, under the Taxpayer First Act of 2019, electronic filing of all Forms 990 will be required for tax years starting July 2, 2021. This change should remediate the known errors from paper-filed returns and increase data reliability. We found several issues with TE/GE’s processing of queries in the Form 990, 990-EZ and 990-PF models that affect the validity or reliability of the scores that the models generate to rank returns for examination selection (see figure 7). As a result, TE/GE cannot ensure that the model scores properly rank the returns for examination selection. Specifically, TE/GE does not consistently assign point values for the queries used to generate the model scores and inform selection decisions. We also found errors in TE/GE’s documentation of the queries, which lead to redundant queries, and inflated model scores. Finally, TE/GE has no control procedures to ensure consistent testing of proposed queries. Inconsistent Point Values for Queries Raise Concerns about Model Scores We estimate that for 24 percent of queries (83 queries) from the models, TE/GE staff did not assign point values for queries consistent with its definitions for the four categories (see table 4). Not implementing the defined point values puts the model scores at risk of inconsistent scoring and examination selection. We found three types of queries involved with the inconsistent assignment of point values. 1. Miscategorized queries were not assigned to the category that matches TE/GE’s definition. These occurred because TE/GE has not documented specific rules for query categorization. As a result, we found an estimated 7.4 percent of queries (26 queries) where TE/GE staff overrode the category definitions when assigning points without documenting the reasons. Absent the reasons, TE/GE cannot ensure consistent treatment of similar queries. In our sample, these override decisions included assigning: Three queries to the Speculative category, which is worth five points, when the definitions supported the Automatic category, which is worth 10 points. TE/GE officials said they did this to offset potentially confusing language in the return lines or instructions. One query to the Automatic category rather than the Speculative category supported by the definitions. TE/GE officials said they used the higher point value category to increase the chance of selection so that certain Form 990-PF attachments, which the queries do not cover, would be more likely to be considered for examination. 2. Queries could fit into more than one category based on TE/GE’s definitions. We estimate that 16 percent (55) of the queries could fit in more than one category. Of these, 18 in our sample could have been placed in the Missing Schedule/Form category. In addition, we found one query in our sample that TE/GE labeled as having a duplicate but one query was assigned to the Automatic category worth 10 points and the other was assigned to the Inconsistencies category worth one point. TE/GE officials acknowledged that some queries could fit in more than one category. When we asked why certain queries for missing schedules and forms were not categorized as such, these officials described a hierarchy of missing forms based on being subject to penalties and interest, such as employment tax returns, and their associated categories. They did not document or consistently implement this hierarchy as queries identifying the same missing form sometimes were in different categories. 3. Sliding scale queries whose point values differ from those stated in TE/GE’s model documentation. We found nine queries with sliding scale point values that involved Form 1099 information returns. The sliding scales reduce point values based on the severity of the compliance issue, such as reducing the query point values if the organization filed a low number of information returns. TE/GE did not provide documentation about the rationale and associated definitions for these queries. Without documentation on the different treatment of these queries, TE/GE is not transparent about the rationale for assigning points through a sliding scale to support its model scoring. TE/GE officials said they have not updated definitions and criteria for using the categories and sliding scales because of a decision to keep the model operating as is and to update documentation as time permits. After our preliminary analyses, TE/GE provided updated definitions for the four categories, and descriptions of the sliding scales that were used for queries. However, these definitions and descriptions do not include any decision rules or criteria that document how to apply them. Further, the sliding scale descriptions do not offer definitions for words like “low,” when referring to the volume of information returns filed. Definitions that are incomplete and not always followed when assigning point values raise concerns about consistency and transparency in scoring returns for examination selection. TE/GE’s assignments affect scores and whether a return is placed on the MSS for examination consideration. Inconsistent or invalid assignment of point values may distort the potential for examination. For example, of the nine miscategorized queries we analyzed in our sample, we determined that if their categorizations were corrected, hits on three of the queries would make a return eligible for the MSS and hits on two others may make a return eligible, depending on the other queries the return hit. Changes to two queries would have made returns no longer eligible for the MSS. Query Documentation Has Errors That Forestall Valid Analysis of Queries We estimate that about 27 percent (96 queries) of the queries in the models had errors in the documented descriptions. Query descriptions detail the logic and data used from specific forms and line numbers that the queries scan. The errors we found include: references to older versions of the forms as well as omissions of form lines used in the query; and query descriptions that did not match programming code. To address these differences, TE/GE proposed corrections to the query descriptions. A TE/GE official said re-visiting the query documentation is part of the contractor’s 2020 study and that TE/GE does not have a timeline for correcting the documentation. In addition to errors, the descriptions also use inconsistent language, which prevents easy identification of queries by issue. For example, to identify all queries related to excess benefit transactions, one must manually search different fields for terms such as “excess benefit,” “excessive benefit,” and “EBT” (excess benefit transaction). Furthermore, TE/GE’s database fields only capture one issue per query. Since many queries involve multiple issues, these fields cannot be used to fully inventory the queries. These errors and inconsistencies in the query descriptions occurred because TE/GE has no procedures for regular reviews of queries as forms or laws change. TE/GE Compliance Governance Board (Governance Board) members review query descriptions prior to implementation but do not review details of the queries in the context of the entire model. Further, TE/GE procedures only require review of programming code before queries are sent to the Governance Board. Review of the code once it is integrated into the model program is optional, according to TE/GE procedures. The errors and inconsistent descriptions prevent TE/GE from having a comprehensive and accurate inventory of queries within and across models. Without regular reviews, TE/GE cannot be assured that its programming code is correct and that any analyses of the performance of queries or the models as a whole are valid. When we asked about the lack of regular reviews of queries, TE/GE officials said they plan to implement reviews but did not provide us with a plan or timeframes for doing so. Another effect of not having a comprehensive and accurate inventory is that TE/GE cannot analyze query performance and identify queries that look for the same compliance issue to prevent redundancies and to ensure valid and consistent scoring. As a result, we found queries that address the same or similar issues with the same criteria, inflating scores for returns and making selection for examination more likely. Our analysis of the July 2019 Form 990 model run showed 90 pairs of queries, involving 78 unique queries that hit together at least 90 percent of the time. By having two queries that rely on the same criteria, returns accumulate extra points for the same behavior. For example, all 910 returns that hit an employment tax query also hit a query that shares some of the same criteria and thresholds. As a result, these returns accumulated 10 points rather than five points, making them eligible for the MSS. Aside from our sample, we found queries seeking certain organizations with political campaign activities and political expenditures that would total 15 points in the Form 990 model. Queries identifying these same activities and expenditures would total 30 points in the Form 990-EZ model. TEGE’s contractor recommended in 2018 that TE/GE eliminate “redundant” queries, which is similar to our finding. TE/GE officials said they do not believe the redundant queries are duplicates and they are awaiting the results of the contractor’s study in 2020 before making changes. Until TE/GE resolves the extent to which it has redundant queries, it cannot do a valid analysis of whether its queries identify the most noncompliant returns. TE/GE Lacks Procedures and Criteria for Testing Proposed Model Queries TE/GE has no procedures requiring the testing of proposed model queries. Even so, based on our sample of the new queries in the fiscal year 2018 Form 990 model, TE/GE would be able to provide evidence of tests for an estimated 94 percent of all new queries. However, TE/GE also does not have procedures for how to conduct testing or what data to use. The testing that has been performed consisted of running the query on certain tax years of returns to count the number of returns flagged, according to a TE/GE official. TE/GE does not run the queries on data from closed examinations to see whether the queries would identify known compliance issues that justify an examination. Interactions with existing queries are not tested. When considering new queries, Governance Board members see the number of returns flagged by each query during testing, but have no criteria to determine whether a query flags an appropriate number of returns. A TE/GE official said TE/GE does not believe it needs to document procedures for testing. In the absence of procedures and standards, TE/GE cannot ensure that testing of new queries is done consistently with appropriate data sources and research standards. By only testing the number of returns that a query flags, TE/GE cannot validate that proposed queries can effectively identify the noncompliance that would be worth examining. Using tested, validated and documented data is a critical step in ensuring that research is proper, reliable, and accomplished in accordance with expectations, according to the IRM. Without testing queries on reliable data, and making adjustments based on criteria, TE/GE risks implementing queries that do not produce reasonable numbers of hits that are worth pursing through examinations. For examination sources that used data other than the models, we found that TE/GE did not always document its processing of data into quality information. We identified common “start-to-finish” segments to this processing of data, including: submitting a proposal and supporting data to find noncompliance; reviewing the potential data sources and queries or thresholds to be used as examination selection criteria; and recommending the proposed effort for approval through the appropriate executives. On one hand, TE/GE provided documentation of the required approvals for these segments in processing data for five compliance strategies. These strategies included examining loans by private foundations and collecting information on organizations that exceed investment income limitations. On the other hand, TE/GE did not provide similar start-to-finish documentation on processing quality information from other examination sources that use data outside of the models; examples include research projects under the Data Driven Approaches portfolio and projects that use queries under the Referrals and Other Casework portfolio (see table 1). Over several discussions, TE/GE did not explain why it did not fully document such projects. By not fully documenting how it processes data into quality information, and by not linking such processes to measurable objectives, TE/GE cannot ensure that it is analyzing quality information in selecting examinations. For the compliance strategies, TE/GE showed evidence of using the quality information to decide which returns to select for examination, such as for Governance Board decisions. However, TE/GE did not provide documentation on how it made selection decisions using data for other projects that use queries (see figure 8). In addition, TE/GE did not use quality information to decide how frequently to run the model. TE/GE decided to run the Form 990 models twice per year without analyzing the effects. Moreover, we found that the time between runs is inconsistent. Since the Form 990 model’s first run, the time between runs ranged from 84 days to 251 days. Since returns are ranked on the MSS, eliminations result in the classification staff selecting lower scoring returns. The average score for examined returns was 27.1 for the list that was used for 84 days compared to 23.2 for the list used for 251 days. To the extent that TE/GE ensures that its model scores are as reliable and valid as possible, analyzing data could help TE/GE identify the frequency of model runs that maximizes the use of model scores to guide decisions on examination selection. For example, analyzing Form 990 filing patterns could help identify the optimal timing of model runs, allowing for adequate time remaining under the statute of limitations. TE/GE does not regularly evaluate its models and other selection processes that use data. In particular, model scores for all returns are not retained or are inconsistent from year to year which limits the ability to conduct evaluations. Furthermore, TE/GE does not evaluate reasons why some selected returns are not examined, which could help improve selection methods (figure 9). TE/GE Has No System for Regularly Evaluating Examination Selection Decisions TE/GE has not regularly evaluated its examination selection decisions that rely on data to improve its selection methods. While TE/GE commissioned the contractor evaluations of its Form 990 model, it has no documented process for continued evaluations of the model or any evaluations of other sources, such as research projects, that rely on data to select returns for examination. For its compliance strategies, not enough examinations have closed under the strategies to warrant evaluations yet, according to a TE/GE official. Data limitations have challenged evaluation efforts, according to a TE/GE official. To address this, TE/GE started capturing more detailed data on examination outcomes; however no evaluations of outcomes have resulted. The officials noted that they have been spending more time reporting and monitoring compared to analyzing and evaluating, which they said needs to occur more often. Without evaluation, TE/GE cannot ensure that its use of data to select returns is working as intended. In addition to not evaluating selection decisions and their outcomes, TE/GE has also not addressed the Form 990 model deficiencies the contractor previously identified. In its 2018 reports, the contractor made 17 recommendations (see appendix V for the status of each recommendation). A TE/GE official said it had not acted on many recommendations because all examination selection strategies are being evaluated with the transition to the Compliance Planning and Classification (CP&C) office. TE/GE initiated another study in 2019 with the same contractor to address its 2018 recommendations among other tasks. As of March 2020, TE/GE implemented one recommendation and part of another, deferred action on nine recommendations until after the contractor finishes the new study, deferred action on three due to other reasons, and did not clearly provide a status for two. In addition, TE/GE will likely not implement the other recommendation. According to contract documentation, the study will explore architectures and alternative designs to the model, propose up to three compliance actions other than examinations, and recommend measures to monitor the actions’ effectiveness. TE/GE expects a final report by September 2020. To the extent that TE/GE has not implemented the contractor’s recommendations from 2018, the related deficiencies identified in the Form 990 model will have persisted for more than 2 years by the time the contractor issues its 2020 report. Unless TE/GE documents its consideration and action of the recommendations, the value of the contractor’s work is diminished and possible improvements may be overlooked. TE/GE Has Not Retained Complete Data to Allow for Full Evaluation of Its Models Until recently, TE/GE did not retain model scores for each return and query performance data that would be useful for evaluation. The January 2018 contractor report recommended that TE/GE save model data. For its July 2018 report, the contractor had to recreate historical scoring data for its evaluation. TE/GE officials said they increased storage space and saved the fiscal year 2018 data. When we asked for these data, TE/GE officials said that each time they run a model, they overwrite the old data. The officials said they did not have space on their server to save all of the data. Instead, TE/GE had been saving the MSS’s for each run. However, the MSSs have only limited value for evaluating the model and queries. Specifically, the MSS for each model run contains score information for only about 20,000 returns (out of about 300,000 scored) that have a certain minimum score and hit queries in certain categories. Further, the MSS does not contain data on model queries that are flagged. In September 2019, TE/GE officials said the Research, Applied Analytics and Statistics division provided temporary server storage space to save model data through September 2020 while the contractor assesses TE/GE’s models. Starting with the July 2019 model run, TE/GE is saving score and query performance data for all filed returns. In January 2020, TE/GE officials told us they developed a way to save data on query hits for all returns run through the model. However, TE/GE has not provided documentation to show exactly what data will be saved over the long term for all filed returns run through the model. Without complete historical data on model scores and query hits, TE/GE cannot assess the full performance of its models. Such data would facilitate an analysis of the queries, and whether they identified returns with changes or related pick- up returns. Historical Data on Examination Outcomes Lack Consistency, Which Complicates Evaluation TE/GE does not analyze consistent multi-year data on examination outcomes, which would facilitate evaluation of its use of data in selecting returns for examination. TE/GE officials said they use historical data— such as change rates—to determine the success of an examination source. TE/GE provided historical data on examination starts, closures, and pick-up returns covering 2 years but did not provide data beyond that and change rates were not always included. Further, TE/GE has used different methods to organize and report examination outcomes over the years. These differences in reporting outcomes affect TE/GE’s data in the following ways: Starting with fiscal year 2018, data on exempt organizations examinations include federal, state and local employment tax examinations. Prior to 2018, TE/GE reported these employment tax examination data separately. After TE/GEs reorganization in 2017, it grouped examinations into portfolios and changed the portfolio definitions during 2018. As of March 2020, TE/GE has not produced a consistent method of summarizing of historical data. TE/GE officials acknowledged data limitations, and said they are working to implement recommendations from a 2019 study to improve capturing examination data. TE/GE officials said the staff member analyzing data has been doing so for many years, allowing them to reconcile the data. However, this poses a risk that other IRS or oversight entities cannot reconcile the data. According to internal control standards, agencies should establish effective methods for retaining organizational knowledge and mitigate the risk of having that knowledge limited to a few personnel, as well as contingency plans to respond to sudden personnel changes. TE/GE’s inconsistent data limit its ability to conduct evaluations. These inconsistent data also prevent TE/GE from establishing baselines or targets for examination outcomes such as change rates to help measure the success of its selection methods. In fiscal year 2019, TE/GE did not examine about 20 percent of the exempt organization returns that had been selected for examination. Although this rate of non-examined returns has improved in recent years, TE/GE has not analyzed data to explore why the rate has improved and how to reduce it further. Our analysis showed that almost 30 percent of these returns were not examined because they were too close to the statute of limitations date. TE/GE officials did not have a reason why the returns were sent to the field for examination if the statute date was so close. TE/GE officials said they do not regularly analyze reasons for non-examined returns. They said they have analyzed only the number of non-examined returns by manager and area. In addition, TE/GE officials said they implemented new guidance in fiscal year 2019 for staff who make decisions to not examine returns, which is intended to improve the information they have on these decisions. As of fiscal year 2019, TE/GE began tracking certain non-examined returns by project code but has not committed to analyzing the data. Non-examined returns are not an efficient use of resources, as the time spent reviewing and rejecting these returns—even if minimal—reduces the time staff have for conducting examinations. Routinely analyzing reasons for non-examined returns, as well as related data, could help TE/GE identify actions to reduce the number of returns that are sent to the field but are then declined for examination by a manager or examiner. TE/GE did not annually update procedures on examination selection and databases in certain IRM sections since the May 2017 reorganization. The Internal Revenue Manual (IRM) states that procedures in IRM sections must be annually reviewed and updated as needed. TE/GE released updated IRM sections for two of the three groups in CP&C. It released a section on the Issue Identification and Research in September 2018, and one on the Classification and Case Assignment procedures in September 2019. However, these sections do not cover the steps the model classifier takes when reviewing returns from the MSS. As of December 2019, no IRM section has been released on the Planning and Monitoring group. As such, TE/GE staff does not always have official information on roles and responsibilities for new entities and processes created since May 2017. For certain updated or new IRM sections, TE/GE did not release interim guidance while those sections awaited approval. IRS requires issuance of interim guidance to address deviations from the IRM, even if temporary. Instead of developing interim guidance, TE/GE officials stated that, in the wake of the reorganization, they decided to use desk guides, such as for the IRM section on classification and case assignment processes. However, TE/GE did not update its desk guides on processes until more than 2 years after the reorganization. Furthermore, the desk guides do not cover the specific duties of the model classifier, or the steps for classification of returns identified for compliance strategies. IRM guidance states that management must develop and maintain documentation on data systems; collection and analysis; and responsibilities for data collection, input and analysis. Timely documentation of new procedures and responsibilities improves the accuracy and reliability of IRM content. According to the IRM, when the IRM and related guidance are not current, TE/GE increases the risk that staff follow incorrect procedures, use guidance that is not transparent to the public, administer tax laws inconsistently, and misinform taxpayers. Good federal government practice requires risk management, without which, TE/GE could undercut its use of data to enhance decisions on examination selection. Although the use of data in examination selection has the potential to improve efficiencies in classifying and examining returns to identify noncompliance, any new endeavor carries risks. TE/GE did not identify any TE/GE-specific risks that could undercut its success in using data to select exempt organization returns for examination. As of December 2019, the TE/GE risk register identified 12 risks, ranging from aging technology and infrastructure to employee engagement and morale. One risk— data access and analytics—involved using data in general decision making at the IRS level rather than TE/GE decisions about examination selection or its related models. TE/GE officials said they are analyzing and responding to this risk under the IRS- wide risk management process. TE/GE did not document why it did not identify any TE/GE-specific risks in using data for examination selection. Our report discusses a number of deficiencies that could be potential risks to TE/GE using data in selecting returns for examination. For example, TE/GE lacks program objectives that would be necessary to identify and assess risks. We also found weaknesses in how TE/GE processes and analyzes data to inform examination selection and how it evaluates selection decisions. Further, the IRM states that TE/GE’s Compliance Governance Board (Governance Board) should consider risks in its decisions and we saw that risks were considered in documents proposing examination selection criteria to the Governance Board. We did not find evidence that TE/GE’s risk management process recorded these risks for analysis and any response if needed. After we shared our concerns about the lack of identified risks, TE/GE officials noted that TE/GE participates in mitigation steps as identified by the IRS Risk Office. TE/GE officials also mentioned CP&C representation in an IRS pilot program designed to explore ways to better select employment tax cases. While such actions could be a component of a risk management strategy, it is incomplete and it is unclear how this initiative would help TE/GE identify, analyze, and mitigate risk. Not identifying and managing risks identified in this report leaves TE/GE open to errors and examination selection decisions that are potentially not transparent or not fair. As such, without objectives and a consistent and documented process for identifying and managing risks, TE/GE cannot effectively address risks that may hamper its efforts to use data to enhance its compliance work. Increasingly constrained resources underscore the importance of TE/GE’s efforts to efficiently identify and examine exempt organization returns that have the highest noncompliance potential. TE/GE has developed ways to use data to aid in examination selection. However, opportunities exist to strengthen internal controls to help ensure that data used are reliable, decision rules are clear and documented, and objectives are identified and being achieved. TE/GE should take several steps to improve the reliability and validity of the models. These steps include improving documentation of decision rules and criteria for scoring; regularly reviewing model documentation and programing; testing new queries and their interaction with existing queries; retaining model and query data; and periodically evaluating the performance of selection methods. In absence of regular evaluation of its examination selection decisions, TE/GE misses opportunities for improving its selection processes. Deficiencies that TE/GE’s contractor already identified provide an opening for improving its models. Without consistent historical data, TE/GE will be limited in assessing progress and making improvements. A review of the reasons why certain returns selected for examination are not examined is an example of an evaluation that could help inform process improvements. Ensuring that all procedures are current and accurate would reduce the potential for employees following incorrect procedures and administering tax laws inconsistently. TE/GE’s lack of identified risks from using data in examination selection precludes TE/GE from analyzing and responding to those risks. By taking actions to further strengthen these internal controls, TE/GE could enhance its efforts to identify and examine the most noncompliant exempt organizations and enhance IRS’s oversight of tax exempt organizations and help maintain the integrity of the charitable sector and the larger exempt community. We are making the following 13 recommendations to IRS: The Commissioner of Internal Revenue should document measurable objectives for using data in selecting exempt organization returns for examination. (Recommendation 1) The Commissioner of Internal Revenue should document and consistently use clear criteria and decision rules on assigning point values to queries, using categories and sliding scales. (Recommendation 2) The Commissioner of Internal Revenue should require a regular review of query descriptions and programming to ensure their accuracy and minimize queries that flag the same or similar compliance issue. (Recommendation 3) The Commissioner of Internal Revenue should develop procedures and criteria to test new queries prior to implementation in the models. (Recommendation 4) The Commissioner of Internal Revenue should more fully document how TE/GE processes data and uses data to make examination selection decisions for sources outside of the model such as research projects and other projects that use queries. (Recommendation 5) The Commissioner of Internal Revenue should conduct an analysis to identify the optimal interval between model runs. (Recommendation 6) The Commissioner of Internal Revenue should establish a process for regularly evaluating selection decisions and related outcomes for the models and other processes that use data to select returns for examinations. (Recommendation 7) The Commissioner of Internal Revenue should document consideration or action on recommendations from its 2018 and 2020 contractor assessments. (Recommendation 8) The Commissioner of Internal Revenue should document how score and query data for all returns in the models will continue to be saved over the long term. (Recommendation 9) The Commissioner of Internal Revenue should ensure that historical data on examination outcomes are consistently defined and used when doing analysis of examination outcomes. (Recommendation 10) The Commissioner of Internal Revenue should routinely analyze the reasons for not examining selected returns and identify any necessary actions to address the reasons. (Recommendation 11) The Commissioner of Internal Revenue should annually review and update procedures as needed in relevant IRM sections on examination selection and issue interim guidance until the affected IRM sections are updated. (Recommendation 12) The Commissioner of Internal Revenue should document why TE/GE has not identified any risks in its risk register for using data to select exempt organization returns for examination. If risks are subsequently identified, TE/GE should document how it plans to analyze and address them. (Recommendation 13) We provided a draft of this report to IRS for review and comment. IRS provided written comments, which are reproduced in appendix VI and summarized below. Of our 13 recommendations, IRS agreed with 12 and disagreed with one. IRS also provided technical comments, which we incorporated as appropriate. IRS disagreed with our recommendation on ensuring that historical data on examination outcomes are consistently defined (Recommendation 10), pointing out that its raw data are consistently defined in its information systems. Our concern, however, is with how the outcome data are reported and analyzed, which inhibits understanding of outcome trends over time. In response to IRS comments, we added language to the final recommendation to more clearly focus on the consistency of the outcome data used and analyzed over the years. In addition, although IRS agreed with our recommendation to more fully document how TE/GE processes and uses data to make examination selection decisions outside of the model (Recommendation 5), IRS said that it would provide documentation on a project (other than compliance strategies) that is approved by the Governance Board. While we look forward to such documentation, we are primarily interested in IRS documenting a system for how it processes and uses data to select returns for examinations for projects outside of the model, regardless of Governance Board approval. As discussed in the report, IRS has such a system for projects in its compliance strategies portfolio, which could provide a framework to follow. Similarly, IRS agreed to analyze return due dates of the filing populations commonly associated with the examinations (Recommendation 6). We will be interested to see how that analysis helps IRS to determine the optimal interval between model runs, which is the focus on our recommendation. We are sending copies to the appropriate congressional committees, the Secretary of the Treasury, the Commissioner of Internal Revenue, and other interested parties. In addition, this report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-9110 or mctiguej@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix VII. This report assesses (1) the use of data to select tax-exempt organization returns for examination; and (2) the process the Tax Exempt and Government Entities (TE/GE) division has established to select returns for examination. To assess the use of data to select tax-exempt organization returns for examination, we reviewed data from the Internal Revenue Service’s (IRS) Returns Inventory and Classification System (RICS) for fiscal years 2016 to 2019. Table 5 defines the variables and measures we analyzed. We analyzed aggregated data at the project code level, and we grouped project codes by examination source (for example, examinations from referrals occurred under several project codes). Based on our testing of the data and review of documentation and interviews, we determined that the data were reliable for purposes of assessing TE/GE’s selection processes. We analyzed outcomes from the Form 990, Return of Organization Exempt from Income Tax model. We used RICS data and Model Score Sheets (MSS), for examinations closed from October 1, 2015 through September 30, 2019. Each model run generates an MSS, which is a ranked list of Form 990s that hit certain types of queries and have a minimum score. We matched Form 990 scores from the MSS with selection information and examination outcomes in RICS for examinations closed under all project codes, though the data presented in objective one is specific to examinations started under the Form 990 project code. We used source codes—which indicate whether an examination was a pick-up, substitute for return or primary return—to analyze what types of examinations produced the highest change rates under the Form 990 model project code. To inform this work, we reviewed recent TE/GE contractor assessments of exempt organization examination selection and the Form 990 model. To assess the process that TE/GE has established to select returns for examination, we reviewed internal controls steps in Standards for Internal Control in the Federal Government (Green Book). Given TE/GE’s emphasis on using data in examination selection, we identified five internal control steps related to analyzing data to select returns for examination to address our objectives. We selected four other internal controls because they constitute practices common to all five steps in the selection process. These are presented in table 6. Define objectives in measurable terms so performance in achieving objectives can be assessed. (Green Book (GB) 6.04) Obtain relevant data from reliable internal and external sources in a timely manner based on identified information requirements. (GB 13.04) Process the obtained data into quality information that supports the internal control system (i.e., using data in decision making); use quality information to achieve the entity’s objectives; and document policies on the responsibilities for data collection, input, and analysis. (GB 13.05, 13.01, and 12.02) Use the quality information to make informed decisions in achieving key objectives. (GB 13.05) Evaluate performance (outcomes) for key objectives and take actions to remediate deficiencies. (GB 13.05, 16.03, and 17.06) Develops, maintains, and updates in a timely fashion documentation on the responsibilities for data collection, input and analysis for using data in decision making. (GB 12.02 and 12.05 and IRM) Defines risk tolerances in specific and measurable terms, considers internal and external factors to identify risks, analyzes risks to estimate significance, and designs specific actions for response. (GB 6.09, 7.04, 7.05, and 7.09) Ensures that personnel possess competence to meet responsibilities as well as understand the importance of effective data analysis in decision making. (GB 4.04) Communicates necessary information to enable personnel to perform key roles for analyzing data in decision making and with external parties. (GB 14.03 and 15.20) To identify criteria specific to IRS, we reviewed the Internal Revenue Manual (IRM), which provides standards and guidance similar to the criteria we identified. We shared the Green Book and IRM criteria with TE/GE, as well as our expectations of the documentation that would show adherence to these criteria. Our assessment focused on examination sources developed after the 2017 reorganization and sources that rely on data for selection (such as models and projects that use queries). Examination sources that did not rely on data, such as claims, were not assessed. We reviewed the referrals classification process to consider how data might be used to enhance it. We analyzed TE/GE documents such as Program Letters, Business Performance Reviews, desk guides, memorandums, work plans, performance data, contractor reports and training documents. In addition, we assessed documents—such as meeting minutes and research results—showing the development and approval of data queries and projects used in examination selection. We reviewed the MSSs for the Form 990 model, and procedures for the Form 990 model, the Form 990-EZ, Short Form Return of Organization Exempt from Income Tax, and Form 990-PF, Return of Private Foundation, models. We selected a generalizable stratified random sample of 114 of the 354 unique queries in the three models (see table 7). Because we followed a probability procedure based on random selections, our sample is only one of a large number of samples that we might have drawn. Since each sample could have provided different estimates, we express our confidence in the precision of our particular sample’s results as a 95 percent confidence interval (e.g., the margin of error is +/- 10 percentage points). This is the interval that would contain the actual population value for 95 percent of the samples we could have drawn. Our sample is designed to control the margin of error of attribute estimates within the overall scope query sample as well as the combined Form 990 query sample (a combination of strata 2 and 4 plus certainty selections). The sample was designed as follows. There is one certainty stratum for Form 990-PF queries where we selected a 100 percent sample (i.e. a census), and this stratum does not have a margin of error. We selected these queries with certainty because of the smaller population size in this stratum. For remaining strata, we selected the necessary sample size to achieve an overall 95 percent confidence interval for attribute (percentage) estimates with a margin of error of about +/-10 percentage points under proportionate allocation. In addition, the sample size was increased in strata 2 and 4 (combining Form 990 model queries) to achieve the necessary sample size for a 95 percent confidence interval with a margin of error of about +/-10 percentage points within this group. For the sampled queries, we compared their category and descriptions as provided in the model documentation, with TE/GE’s definitions of the categories to assess whether the query was categorized appropriately. We also compared the query descriptions with the forms to assess whether the referenced lines were relevant to the query. Additionally, we reviewed the model programming code to check for errors and consistency with the query descriptions. For query categorizations that did not match TE/GE’s definitions or queries that appeared to have errors in the descriptions or programming, we asked TE/GE to review and explain its decisions. To identify potentially redundant queries, we analyzed output from the July 2019 Form 990 model run, the only one available at the time of our analysis. Within our sample, we reviewed 36 of the 104 newly added queries in the fiscal year 2018 model. Specifically, we reviewed approval documentation and meeting minutes to test whether two levels of management and the Compliance Governance Board approved new queries, consistent with TE/GE procedures. We also reviewed evidence that TE/GE tested each query prior to its approval for inclusion in the models. We held two telephone focus groups with the nine classifiers who review exempt organizations referrals. We asked questions about the data and resources they use to classify referrals, how they convey their results, and how they are provided feedback. We interviewed officials from the Compliance Planning and Classification office and IRS’s Research, Applied Analytics and Statistics division who worked on several compliance research initiatives. We met regularly with TE/GE to share ongoing assessments. We conducted this performance audit from November 2018 to June 2020 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. The figure below shows the text of the 2019 version of Form 990, Return of Organization Exempt from Income Tax. A list of schedules for the Form 990 is provided in table 8 following the form. The remaining pages of the Form 990 are available at IRS’s website, accessed March 23, 2020: https://www.irs.gov/pub/irs-pdf/f990.pdf. Most exempt organizations are required to file an annual form to report their activities, structure, revenue and expenses, and other items. The organization’s classification under the Internal Revenue Code and its gross receipts and total assets, determines which form must be filed. Most organizations file one of the following: Form 990, Return of Organization Exempt from Income Tax; Form 990-EZ, Short Form, Return of Organization Exempt from Form 990-PF, Return of Private Foundation or Section 4947(a)(1) Trust Treated as Private Foundation. The Internal Revenue Service (IRS) last redesigned the Form 990 series for tax year 2008. The redesign added 14 schedules to the existing two, and reflected changes in the tax-exempt sector and tax law. Some changes from the redesign were phased in and implemented for tax year 2008 and 2009 filings. We summarized changes as found in the “What’s New” section of the form instructions for each of the three Form 990 types and for each year. We grouped the changes into two categories as defined by: New or revised question(s): The addition of new lines, check boxes, narratives or schedules. This includes changes to accommodate new laws or reporting requirements, such as new reporting thresholds or standards. Instructions and format: New descriptions or details in the instructions, such as specifying examples or how to provide certain information to IRS. This also includes changes that affect order of lines or schedules, but not the content. For the Form 990 changes since the redesign, IRS made 56 changes to the form or its instructions for tax years 2009 through 2019 (see table 9 below). These changes include three to the 2018 form implementing new excise taxes on net investment income of certain colleges and universities and on certain tax-exempt organization executive compensation. Aside from new electronic filing requirements for tax years beginning July 2, 2019, the 2019 form did not have any changes. In addition to the 56 changes, IRS made 95 clarifications to existing lines or instructions, or revisions to definitions from tax years 2009 through 2018. These clarifications provide more specific definitions or other details. Further, several of the schedules had additions. For example, the Patient Protection and Affordable Care Act led to additional reporting on Schedule H, Hospitals, to fulfill requirements that hospitals report on each of their facilities and conduct a Community Health Needs Assessment every 3 years. Most of the Form 990-EZ’s 27 changes occurred in tax years 2009 through 2012, of which 12 were for 2011 and several of them focused on compensation reporting. IRS also made 27 clarifications for 2009-2013. Public Law 115-97 did not affect Form 990-EZ. There were no changes to the 2019 form. See table 10. For the Form 990-PF, IRS made the fewest changes compared to Forms 990 and 990-EZ, with only 11 changes and four clarifications for tax years 2009 through 2019. The Form 990-PF had three changes prompted in 2018 by Public Law 115-97. Electronic filing requirements apply to Form 990-PF for tax years starting July 2, 2019, but there were no other changes for the 2019 form. See table 11. Appendix IV describes the general examination selection process for exempt organization returns, and specific classification steps that apply to certain returns. The annual work plan is the foundation for identifying and assigning returns for examination. The Compliance Planning and Classification (CP&C) office follows various steps to identify returns to fulfill the work plan, which end in the assignment of returns for potential examinations to field work groups. The intended process is in figure 11 and discussed below. Annual work plan. CP&C’s Planning and Monitoring group develops the annual work plan. The work plan provides estimates of examination starts and closures. It also has estimates for the number of hours to be spent per return examination and the number of days to complete an examination. Planning and Monitoring develops estimates at the project code level, which corresponds to a specific examination source or project such as the Form 990 model. The Tax Exempt and Government Entities’ (TE/GE) Compliance Governance Board approves the work plan. TE/GE provides a summary of the work plan in its annual Program Letter. Stocking report. The Planning and Monitoring group uses the work plan to issue “stocking” reports to guide classifiers on types of returns to identify for potential examination. Planning and Monitoring considers available examiners, and progress in meeting work plan numbers. The report lists the number of returns needed by grade, project code, and classification source. Classification. Classifiers review stocking plans to identify returns for potential examination. Classifiers are to eliminate returns for consideration if the (1) return is approaching its statute of limitation date, (2) organization has been examined in the last 3 years, or (3) organization is under a compliance check. Establishing the return and initial case building. If classifiers identify examination potential, they establish returns in the Audit Information Management System and Reporting Compliance Case Management System (RCCMS). The returns are sent for initial case building—developing paperwork to initiate the examination— according to a TE/GE official. Virtual shelf. Established returns and the initial case material are sent to the virtual shelf, which is an electronic inventory of returns that may be assigned for examination. Certain referrals, claims, compliance strategies, and other returns are prioritized, according to a TE/GE official. Returns remain on the shelf until assigned for examination or otherwise closed due to statute of limitations, according to a TE/GE official. Examination assignment. Functional Assignment Coordinators pull returns from the virtual shelf to fulfill field group work requests. Returns on the virtual shelf that matched a work order undergo additional case building before delivery to field examination groups. Monitoring. Planning and Monitoring staff regularly review reports that compare work plan goals with current work, and run algorithms to forecast upcoming work. These reviews are intended to ensure that sufficient work is available for assignment, excess work is not created, and returns approaching statute of limitations are identified. The monitoring informs new stocking reports. Classification steps vary depending on how a return was identified for potential examination. For returns identified with queries or models, classifiers check a limited set of criteria once a return is identified. For returns identified through other sources, such as referrals, the classifier also reviews facts and circumstances about potential noncompliance in returns. We focus here on examination sources that rely on data—such as models or queries—and referrals. Referrals are complaints of exempt organization noncompliance made by third parties, including the public and other parts of the Internal Revenue Service. We describe referrals classification because it is one of the top sources of exempt organizations examinations. The models are run to identify returns with potential noncompliance and lists them on a Model Score Sheet (MSS). The MSS is a ranked list of returns by scores from the model. According to a TE/GE official, the classifier: works down the list, starting with the highest scores, to fill stocking checks whether the return was also identified for a compliance strategy; and. eliminates returns based on the statute of limitations and recent examination activity. For some projects in the Compliance Strategies portfolio, a query is run or returns are sampled to identify a population meeting indicators of potential noncompliance. Then, the classifier uses the stocking report to select returns with certain geographic or case grade criteria and eliminates returns based on statute of limitations, recent examination status, and resolving non-filing issues, according to a TE/GE official. TE/GE classifiers do a triage to review and eliminate referrals that are not relevant to tax administration or do not have substantiated information. The triage classifier sorts referrals and reviews the following: organization status (for example, already revoked or terminated); examination history of the organization; and evidence of substantial inurement or private benefit, non-exempt activities, or material employment tax or unrelated business income that would result in a significant tax assessment. Referrals that pass triage are either sent to classification or, if they deal with political issues, are sent to a committee of three TE/GE managers, who vote on a selection decision. For all referrals, the classifier researches the referral. Research sources include websites, external databases, and IRS taxpayer account databases. The classifier may look at the organization’s website, information about officers, or prior examination history. Referrals with examination potential are either assigned immediately or placed on the virtual shelf. Referrals that must be immediately assigned include those with strong indicators of fraud, illegal or illicit activities (including terrorism), or referrals from whistleblowers, or certain other IRS divisions. Other referrals are labeled as high, medium or lower priority, based on potential for revocation or significant tax assessments. The Tax Exempt and Government Entities division (TE/GE) hired a contractor to review the effectiveness of its Form 990 examination selection model. The contractor prepared two reports. The first, delivered in January 2018, makes recommendations on the model process, the computing environment, and performance measures. The second, delivered in July 2018, makes recommendations to more effectively and efficiently identify returns for examination, such as through the model. Within the two reports, the contractor made 17 recommendations. Table 12 lists the recommendations and the status of each. In September 2019, TE/GE initiated another study, anticipated to be completed in September 2020. The study focuses on developing alternatives to enhance the models. The study will explore architectures and alternative designs for the model and propose alternative compliance actions to examinations and recommend measures to monitor their effectiveness. In addition to the contact named above, Tom Short (Assistant Director), Lindsay Swenson (Analyst-in-Charge), Ann Czapiewski, George Guttman, Amalia Konstas, Krista Loose, Alan Rozzi, Cynthia Saunders, Andrew J. Stephens, and Sonya Vartivarian made key contributions to this report.

Summary: Exempt organizations often provide charitable services, or in some instances, membership benefits in furtherance of an exempt purpose. They generally do not pay federal income tax. IRS examines exempt organization returns (Form 990 and others) to address noncompliance, which may promote confidence in the tax exempt sector. In 2016, IRS started using three analytical models using Form 990 data to identify potential noncompliance and select returns for examination. GAO was asked to review IRS's use of Form 990 data. This report assesses (1) IRS's use of data to select returns for examination and, (2) the process IRS has established for selecting returns. GAO analyzed (1) examination data from fiscal years 2016 through 2019 including results from the largest Form 990 model, and (2) model documentation for a generalizable sample. GAO interviewed IRS officials and assessed IRS policies and procedures using relevant standards for internal control. The Internal Revenue Service (IRS) used data to select almost 70 percent of its examinations of Form 990 returns in fiscal year 2019. Almost half of these examinations were selected using models that score returns for potential noncompliance (see figure). Of the returns examined that were selected using the model, 87 percent resulted in a change to the return, indicating that IRS identified noncompliance. GAO found that the model did not improve change rates compared to prior selection methods and a higher model score is not associated with a higher change rate. IRS has not fully implemented or documented internal controls in its established processes for analyzing data for examination selection. For example: IRS has not defined measurable objectives for using data to select returns for examination . Without measurable objectives, IRS cannot assess how well it is doing or fully implement other internal controls. IRS's models have deficiencies affecting the validity and reliability of return scoring and selection . IRS has incomplete definitions and procedures and did not always follow its definitions when assigning point values for identifying potential noncompliance for examination. As a result, return scoring by the models is not always consistent. IRS did not consistently document the processing and use of data in decision-making on examination selection . Without such documentation, IRS cannot support its use of data in examination selection in all cases. IRS does not regularly evaluate examination selection. Examination data were inconsistent across years and IRS only tracks one prior year of data. IRS also did not save data on all returns that the models scored. Without data and regular evaluations, IRS cannot assure that its models are selecting returns as intended and that deficiencies are identified and corrected. GAO makes 13 recommendations, including that IRS establish objectives, revise model documentation, fully document processing and using data in decisions, and regularly evaluate examination selection. IRS agreed with all recommendations except one related to evaluating examination selection methods using consistent historical data over time. GAO continues to believe that this recommendation is valid as discussed in the report.