The MTW demonstration was authorized by the Omnibus Consolidated Rescissions and Appropriations Act of 1996 (1996 Act). The demonstration’s ultimate goal is to identify successful approaches that can be applied to public housing agencies nationwide. As of November 2017, a total of 39 agencies were authorized to participate in the demonstration (see fig. 1); however, two agencies consolidated their MTW demonstration programs and are counted as one agency for purposes of MTW participation. The MTW Office within the Office of Public and Indian Housing (PIH) is responsible for implementing the demonstration. The MTW Office currently includes a program director and eight coordinators, who are each assigned to a specific group of MTW agencies. MTW coordinators facilitate the reviews of planned and implemented activities and are responsible for coordinating with other HUD offices, including local HUD field offices, to obtain additional input on MTW agencies’ planned activities and accomplishments. The 1996 Act that created the MTW demonstration provides three objectives for the demonstration: (1) reduce costs and achieve greater cost-effectiveness in federal housing expenditures; (2) give incentives to families with children where the head of household is working, seeking work, or is preparing for work by participating in job training, educational programs, or programs that assist people to obtain employment and become economically self-sufficient; and (3) increase housing choices for low-income families. In making these changes, MTW agencies must comply with the following five contractual requirements derived from the 1996 Act: 1. assist substantially the same total number of eligible low-income families under MTW as would have been served absent the demonstration; 2. maintain a mix of families (by family size) comparable to those they would have served without the demonstration; 3. ensure that at least 75 percent of the families assisted are very low- 4. establish a reasonable rent policy to encourage employment and self- 5. assure that the housing the agencies provide meets HUD’s housing quality standards. MTW agencies do not receive special funding allocations; rather, they receive funds from the three traditional primary funding sources (public housing capital funds, public housing operating funds, and voucher funds). Public housing agencies generally are required to use the funds from each source only for specific purposes, but MTW agencies may combine the money from the three sources and use the funds for a variety of HUD-approved activities. This fungibility is intended to give MTW agencies greater flexibility. For example, public housing operating funds are traditionally used to make up the difference between rents charged for units and the cost of operating them. Capital funds are traditionally used for modernization and management improvements, while voucher funds traditionally provide rental assistance in the private market. However, an MTW agency may use public housing capital funds to issue additional vouchers or use voucher funds to develop more public housing. MTW agencies also have the authority to use their funds to implement innovative activities that differ from traditional housing assistance. For instance, an MTW agency can use funds to replace public housing with mixed-income communities or reach special-needs populations using vouchers paired with supportive services. HUD entered into a standard agreement with each existing MTW agency. HUD created the agreement in 2008 to standardize the contract terms. The agreement references an attachment that sets out reporting requirements (Attachment B or Form 50900) and another attachment (Attachment C) that lists the specific sections of the United States Housing Act of 1937, as amended, and its implementing regulations that an MTW agency may waive as part of its MTW flexibility. While the standard agreement is generally the same for all MTW agencies, two attachments are tailored to individual agencies: a description of the formulas for determining the amounts of funding each agency will receive (Attachment A) and a section that may include some agency-specific authorizations (Attachment D). In addition to statutory requirements, the agreement requires all existing MTW agencies to submit to HUD an annual plan for approval as well as an annual report. Attachment B outlines the information that agencies are required to include in their annual plans and annual reports. For example, MTW agencies must include certain elements in their annual plans for each activity they propose to adopt, such as (1) a description of the activity and its anticipated effect in relation to the statutory objective under which the activity is proposed; (2) the HUD metrics that will be used to quantify the changes the agency anticipates as a result of the activity, including baseline performance level and yearly benchmarks; and (3) the MTW authorizations that give the agency the flexibility to conduct the activity. Similarly, MTW agencies are required to include in their annual reports information about housing stocks and leasing as well as information required for HUD to assess compliance with key demonstration requirements (such as number and mix of families served and percentage of very low-income households served). For rent-reform activities, agencies are also required to describe the number and results of any hardship requests. MTW agencies also are required to report standard information through HUD data systems. MTW agencies must submit tenant-related data into the Moving to Work section of the Public and Indian Housing Information Center (MTW-PIC). According to HUD officials, the MTW-PIC module was created in 2007 because the standard PIC system that non-MTW agencies use could not accommodate some of the activities allowed under MTW, such as rent calculations that vary from HUD’s standard calculations. MTW agencies also must submit year-end financial information into FDS, and HUD issued special instructions to enable MTW agencies to complete the reporting. Finally, MTW agencies must report voucher unit utilization in VMS. The Consolidated Appropriations Act, 2016 authorized HUD to expand the MTW demonstration from the current 39 public housing agencies to an additional 100 agencies (expansion agencies) over 7 years. The 2016 act requires that the expansion agencies must be high performers at the time of application and that the selected agencies represent geographic diversity. The expansion agencies will be brought into the demonstration by cohort, as required by the 2016 act. HUD plans to designate the initial cohort by summer 2018. As directed by the 2016 act, within each cohort each agency will implement one policy change that HUD selects for that cohort to test. The 2016 act requires that expansion agencies be rigorously evaluated and that HUD establish a research advisory committee to advise the Secretary on policies to study and methods of research and evaluation. HUD established the committee and received its recommendations on which policy changes to test and how to evaluate them. As of November 2017, HUD had not announced the policy changes each cohort will be testing. On January 23, 2017, HUD published in the Federal Register a request for comment on a draft operations notice for the MTW expansion. The draft operations notice establishes requirements for the implementation and continued operations of the demonstration and describes waivers available, terms of participation, funding and financial reporting, and administration and oversight for agencies joining under the expansion. The comment period closed on June 5, 2017. According to HUD officials, there will be another opportunity for comment before the notice is finalized in early 2018. Since our last review of the MTW demonstration in April 2012, HUD has taken steps to improve MTW agencies’ annual reporting and its process for monitoring agencies’ compliance with requirements of the demonstration. However, we found that HUD’s oversight—review of annual reports and compliance assessments—has not been timely and HUD has not fully documented its process for assessing compliance, due to limited staffing and competing priorities. While the MTW Office added staff to assist with the oversight of the current 39 MTW agencies, HUD has not conducted workforce planning to address the resources needed for overseeing the 100 agencies to be added through the MTW demonstration expansion. HUD has taken steps to improve MTW agencies’ annual reporting. While agencies were already required to submit annual plans and reports, HUD revised its reporting requirements for MTW agencies in May 2013 in response to our recommendations. Specifically, HUD revised Attachment B to provide detailed clarifications on the meaning of the three statutory objectives of the demonstration and relevant standard metrics. For example, for each of the statutory objectives, the revised guidance requires that the MTW agency use and report on all of the applicable standard metrics listed in Attachment B. The revised attachment also includes standard tables for MTW agencies to provide operating information and financial information. Additionally, HUD conducted training on the revised Attachment B and issued a document containing answers to frequently asked questions about the revisions. HUD also took some steps to improve its monitoring of MTW agencies’ compliance with the five requirements of the demonstration. Specifically, in response to our 2012 recommendation that HUD implement a process for assessing compliance with the requirements, HUD developed a process and began to track MTW agencies’ compliance with each of the five requirements. The 2013 revisions to Attachment B added requirements for agencies to submit information in annual reports with which HUD assesses compliance. The attachment includes standard tables for MTW agencies to provide specific information on households served, family sizes, and income levels. According to our review of HUD documents and discussions with HUD officials, the MTW Office uses this information, along with information MTW agencies submitted in other HUD data systems, to assess compliance with the five requirements. Table 1 summarizes HUD’s description of its compliance processes for each of the five requirements. We found that HUD’s reviews of MTW agencies’ annual reports were not completed in a timely manner; reviews were completed multiple years after the annual reports were submitted. Specifically, HUD did not complete its review of the agencies’ 2013–2015 reports until March 2017 and its review of 2016 reports was still underway as of November 2017 (see fig. 2). As previously mentioned, MTW agencies submit information about their MTW activities, financial information, data related to compliance assessments, and other information through annual reports. Attachment B states that HUD officials will use this information to monitor MTW agencies, particularly their compliance with some of the five requirements. Although the standard agreement gives MTW agencies 90 days after the end of their fiscal year to submit the annual report to HUD, it does not specify a time frame for HUD’s review of the report. However, it states that HUD must notify an agency in writing if it requires additional information or clarifications to the information provided in the report. HUD officials said that limited staffing resources in the MTW Office in 2014–2016 led to delays in the reviews. Officials further noted that in 2014 and 2015 existing staff in the MTW Office had to focus on other priorities, including renegotiating the standard agreement, and then in 2016 on implementing the expansion of the demonstration. Untimely reviews of MTW annual reports diminishes oversight and can result in delays on HUD’s part in responding to issues arising from the review, agencies not having an opportunity to respond to concerns promptly, and HUD’s inability to assess the information reported to determine effects on tenants. As previously described, HUD developed a process to assess compliance with the five requirements of the demonstration, but its implementation of the process was not always timely. HUD did not complete its 2013–2015 reviews of MTW agencies’ compliance with the five requirements until 2017. In March 2016, HUD officials provided us with a tracker of agencies’ compliance with the requirements that indicated HUD started its review for 2013 but had not yet completed that assessment or started reviewing compliance for subsequent years. In July 2017, HUD provided us with evidence it had completed the 2013–2016 assessments for all five requirements. In addition, HUD has not clearly documented its process for assessing compliance with the five requirements. HUD officials told us they did not have documentation of the process they used to assess compliance with most of the requirements, such as the methodologies and data used. As previously discussed, HUD has different processes for assessing compliance with each requirement and the information it uses to determine compliance comes from various data sources. Based on our review of HUD documents (including Attachment B and the recently completed compliance assessments) and discussions with HUD officials, it was not always clear what methods HUD used to support its compliance determinations. For example, documentation we reviewed on the requirement that MTW agencies ensure that 75 percent of the households served are very low-income did not state the methodology used to determine if MTW agencies were in compliance. While our review of the documentation indicated that tenant income in all relevant programs was used, it was not clear if the percentages of tenants in each income category were averaged or weighted to obtain the final percentage of tenants with very low incomes. Additionally, while Attachment B briefly describes the data sources used for some of the compliance assessments, HUD has no documentation specifying what data variables to extract and how to use them. The lack of written instructions led to HUD having to redo its assessment of compliance with the requirement that MTW agencies ensure that 75 percent of the households served are very low-income. Specifically, HUD officials noted that HUD staff initially determined compliance with this requirement based on tenants’ current income, but later determined that they needed to reassess compliance with the requirement using tenants’ income at the time of entry to the program. In September 2017, HUD officials said they were developing internal standard operating procedures to document their approach to assessing compliance with each requirement, and expected to complete the procedures by early calendar year 2018. However, because HUD has not finalized these standard operating procedures, it is unclear whether they fully document the steps and data needed to complete the compliance assessments. Federal internal control standards state that management should develop and maintain documentation of its internal control system, including for controls related to any compliance objectives of the agency. They note that effective documentation assists in management’s design of internal control by establishing and communicating purposes, roles and responsibilities, and specifics of implementation to agency staff. HUD officials stated that limited staffing in the MTW Office in 2014–2016 and competing priorities led to delays in compliance assessments and development of full documentation on procedures. Limited documentation for assessing compliance can lead to inconsistent monitoring of MTW agencies’ compliance with the five requirements. For example, as previously discussed, the lack of documentation on the process and data needed led to the need to reassess compliance with the requirement that MTW agencies ensure that 75 percent of the households served are very low-income. While HUD has taken some steps to address oversight and staff responsibilities for an expanded demonstration, it has not conducted workforce planning for the expanded demonstration. Federal internal control standards state that management should design control activities, including management of human capital, to achieve objectives and respond to risks. Management is to continually assess the knowledge, skills, and ability needs of the entity so that the entity is able to obtain a workforce that has the required knowledge, skills, and abilities to achieve organizational goals. In previous work on human capital, we identified key principles for effective strategic workforce planning, including determining the critical skills and competencies needed to achieve current and future programmatic results and developing strategies that are tailored to address gaps in number, deployment, and alignment of human capital approaches for enabling and sustaining the contributions of all critical skills and competencies. In 2014, the MTW Office engaged in a workforce analysis exercise to determine staffing levels needed to oversee the MTW demonstration as configured at that time. Based on the 2014 analysis, the MTW Office determined that seven staff were needed to oversee the 39 participating agencies. In 2014, the MTW Office had four staff and in 2015, five (see table 2). Officials told us that in 2016, an additional five staff were hired in the MTW Office and that one staff member would focus on financial analysis and compliance assessment. In 2017, the MTW staff count was nine. In July 2017, officials told us that based on the 2014 workforce analysis, they determined they had sufficient resources to oversee the current 39 MTW agencies. In response to a congressional request to determine resource needs for MTW expansion, in December 2015 the MTW Office updated its 2014 workforce analysis. As with the 2014 analysis, the 2015 workforce analysis discussed the level of staffing resources needed and not the skill sets and competencies needed to oversee the expanded MTW demonstration and actions to fill any gaps. According to this analysis, HUD determined that a significant number of staff would be needed to oversee the new agencies. Specifically, 41 full-time equivalent personnel across various HUD offices would be needed to meet the resource needs of the expansion in 2016–2020. In September 2017, HUD officials said that because of the current budget environment, the agency planned to address the staffing gap identified in the 2015 analysis by developing a joint oversight structure between the MTW Office and PIH’s Office of Field Operations. According to HUD officials, currently the MTW Office is primarily responsible for monitoring MTW agencies (reviewing annual plans and reports and assessing compliance with demonstration requirements). Field office staff in PIH assist with the review of MTW agencies’ overall financial health and public housing occupancy and voucher leasing information, among other things. HUD plans to continue to follow this oversight structure for the existing 39 agencies, but have field office staff assume more responsibilities for agencies that will join the MTW demonstration as a result of the expansion. MTW Office officials said they have been having internal discussions through a working group with field office staff in PIH to discuss the new oversight structure and determine how best to meet resource needs associated with the expansion. However, as of November 2017, the MTW Office and PIH had not completed plans for joint oversight of the expanded MTW demonstration with the field offices or assessed the knowledge, skills, or abilities needed to implement this new oversight structure. As previously stated, the first cohort of public housing agencies will join the expanded MTW demonstration by summer 2018. MTW Office officials also told us that PIH is planning to finalize a workforce plan by early calendar year 2018 that will address the broad resource needs of PIH. However, according to MTW Office officials, PIH has not yet determined the extent to which the human capital resource needs for the MTW expansion will be incorporated into the PIH workforce plan. Without strategic workforce planning that reflects the oversight strategy for the expanded MTW demonstration, identifies the critical skills and competencies needed, and includes strategies to address any gaps, HUD will not be able to reasonably ensure that it has the staffing resources necessary to oversee an expanded demonstration. We found significant differences between MTW agencies and comparable non-MTW agencies in key outcomes: MTW agencies had lower public housing occupancy rates, lower voucher unit utilization rates, and higher program expenses in 2009–2015 than similar non-MTW agencies. MTW funding flexibilities may partly explain the differences, but limitations in HUD data (such as the inability to determine which funding source was used to fund which activity) make it difficult to more fully understand the differences. MTW agencies accumulated relatively large reserves of voucher funding, but HUD has performed limited oversight of reserves for these agencies. We found significant differences between MTW agencies and comparable non-MTW agencies in key outcomes of the public housing and voucher programs, possibly affecting the number of tenants MTW agencies served. MTW agencies had lower yearly median public housing occupancy rates in fiscal years 2009–2015 than comparable non-MTW agencies, and the difference was statistically significant (see fig. 3). The median share of public housing units occupied (public housing occupancy rate) for MTW agencies was 3 percentage points lower than for similar non-MTW agencies (93 versus 96 percentage points). The middle 50 percent of MTW agencies in our analysis had occupancy rates that ranged from 88 to 96 percentage points, while the non-MTW agencies in our analysis had occupancy rates that ranged from 92 to 98 percentage points. MTW agencies also had lower rates of voucher unit utilization than comparable non-MTW agencies in each year during 2009–2015 (see fig. 4). The voucher unit utilization rate for MTW agencies was about 3 percentage points lower than for similar non-MTW agencies (about 93 percent versus about 96 percent). The middle 50 percent of the MTW agencies had utilization rates that ranged from about 82 to 97 percentage points, while the non-MTW agencies had occupancy rates that ranged from about 92 to 98 percentage points. We also analyzed expenses for the public housing and voucher programs of MTW agencies and comparable non-MTW agencies in 2009–2015. For the public housing program, we included all operating expenses the MTW and non-MTW agencies incurred that were associated with their public housing properties. As figure 5 shows, median public housing operating expenses for MTW agencies in each year during 2009–2015 were $7,853 per household and $6,622 for non-MTW agencies, a difference of about 19 percent. The middle 50 percent of the MTW agencies had total public housing expenses that ranged from $6,048 to $11,436, while the non-MTW agencies had expenses that ranged from $5,827 to $8,355. We also compared the operating expenses associated with the central office cost center of MTW and comparable non-MTW agencies. If larger public housing agencies implement HUD’s property management rules, they generally are required to create a central office cost center, which manages all the centralized activities of the agency and earns fees for providing day-to-day oversight of individual public housing properties such as property management. As figure 6 shows, median public housing operating expenses related to the central office cost center for MTW agencies were about 9 percent higher than comparable non-MTW agencies in each year during 2009–2015 ($2,745 per household and $2,520, respectively). The middle 50 percent of the MTW agencies had central office cost center expenses associated with their public housing program that ranged from $1,509 to $5,798, while the non-MTW agencies had expenses that ranged from $1,635 to $4,939 per household. For the voucher program, we separately examined expenses in 2009– 2015 related to administration, subsidy (housing assistance payments), and tenant services. MTW agencies had higher median administrative, subsidy, and tenant services expenses than comparable non-MTW agencies. As figure 7 shows, median yearly administrative expenses for MTW agencies were $922 per household and $642 for comparable non- MTW agencies, a difference of about 43 percent. The middle 50 percent of the MTW agencies had voucher administrative expenses that ranged from $713 to $1,179, while the non-MTW agencies had expenses that ranged from $555 to $762. As shown in figure 8, the yearly median voucher subsidy expenses for MTW agencies were about 25 percent higher than for comparable non- MTW agencies ($8,295 per household for MTW agencies and $6,629 per household for non-MTW agencies). The middle 50 percent of the MTW agencies had voucher subsidy expenses that ranged from $6,128 to $12,201, while the non-MTW agencies had expenses that ranged from $5,524 to $8,178. As shown in figure 9, the tenant services expenses for the voucher program were higher for MTW agencies than for comparable non-MTW agencies, and many non-MTW agencies did not record any expenses for tenant services in HUD’s database for the years we reviewed. These results are consistent with MTW agencies having more flexibility to use funds to provide tenant services. The median yearly expenses for tenant services for MTW agencies were about $37 per household. Although tenant services are an allowable administrative expense under the traditional voucher program, more than half of the non-MTW agencies in our sample did not report any expenses for tenant services for most of the years we examined. Non-MTW agencies generally use their voucher funds to make subsidy payments to landlords and for administrative expenses. The statistical matching and modeling analysis we conducted improved upon unadjusted comparisons of MTW and non-MTW agencies, but it was not designed to estimate the causal effects of MTW flexibilities. To reduce the influence of known differences between the two groups, we accounted for broad characteristics that differed between MTW agencies and non-MTW agencies. However, our analysis did not attempt to measure the unique circumstances of each MTW agency, but rather broad outcomes relevant to public housing and voucher programs in general. For additional details on our methods and results, see appendix II. As noted by others who studied the MTW demonstration and our previous report, no central source of systematic data exists for MTW activities and outcomes. However, a July 2017 report by Abt Associates, a research and consulting firm, identified and tested indicators they developed to track the performance of MTW demonstrations and compare them to similar non-MTW agencies. As with our analysis, the Abt study found MTW agencies tended to have worse outcomes than similar non-MTW agencies on the indicators of voucher administrative expenses and voucher unit utilization. The study also analyzed other indicators such as increases in earnings of nonelderly, nondisabled households; households served by a service coordinator; and share of voucher households in neighborhoods with lower poverty rates. On many of the other indicators analyzed, the study found that MTW agencies did better than similar non- MTW agencies. For example, for the self-sufficiency measures examined in the study, estimates showed that household earnings were more likely to increase at MTW agencies than at comparison non-MTW agencies. The study also concluded that MTW agencies were able to serve a significant number of individuals not reached by traditional housing assistance and that in many cases, they were also able to offer additional supportive services. However, because our analysis did not look at these other indicators, we could not confirm these results. The observed differences in public housing occupancy and voucher unit utilization rates and program expenses between MTW and non-MTW agencies, which could affect the number of tenants served, may be a result of MTW agencies’ ability to (1) combine their public housing and voucher funds and use them interchangeably and (2) use funds to implement policies that go beyond traditional forms of housing assistance. Combined funding and fungibility. The single fund authorization permits MTW agencies to combine their public housing operating, public housing capital, and voucher funds into a single agency-wide funding source and use the funds interchangeably. For instance, voucher funds may be used for public housing expenses and vice versa, which could affect utilization and occupancy rates. Our analysis of 2015 data from FDS, which HUD uses to account for the agencies’ MTW financial data, showed that 19 MTW agencies transferred voucher funding to their public housing program as the result of the single-fund authorization (that is, they transferred more funding to their public housing accounts than they received through their public housing funding allocation). This analysis was possible because HUD requires agencies to report financial information in FDS at the public housing project level. However, the data could not be used to determine whether all the funds transferred to the public housing accounts were spent on public housing expenses because, according to HUD officials, FDS is not a system that tracks the actual drawdown or disbursement of funds. Instead, public housing agencies use the system to report year-end financial activity. (As discussed later in this report, FDS data could not be used to determine the extent to which public housing funds were used for voucher expenses.) Nontraditional activities. Public housing occupancy and voucher unit utilization rates might be lower for MTW agencies in part because MTW agencies can use funds to implement policies that go beyond traditional forms of housing assistance. Since October 2009, the demonstration’s “broader uses of funds” authorization under the standard agreement has permitted all MTW agencies to adopt local, nontraditional activities, which HUD guidance organizes into four categories (see table 3). In July 2017, HUD provided us with data it had recently compiled on the number of households served through local, nontraditional activities, by MTW agency, during 2009–2016 (see fig. 10). According to these data, in 2009 four agencies implemented at least one type of local, nontraditional housing assistance activity and served 1,177 households (that is, less than 1 household served through local, nontraditional housing assistance for every 100 MTW public housing and voucher units available). In 2016, the number of agencies that implemented at least one local, nontraditional housing assistance activity grew to 25 agencies, which served 9,787 households (about 2 households served through local, nontraditional housing assistance for every 100 MTW public housing and voucher unit available). Some of these households could be served through a rental assistance program that offers a lower level of subsidy than is available to households served through traditional voucher and public housing programs. For example, a local, nontraditional activity could result in an MTW agency lowering its share of housing assistance, thereby increasing the tenant’s share of rent. Conversely, HUD officials pointed out that because MTW agencies assist hard-to-serve households, the subsidies provided to these households could be higher than the subsidy provided under HUD’s traditional housing assistance programs. As such, a household served through local, nontraditional housing activity may not be equivalent to a household served under the traditional voucher or public housing program. Other factors related to expenses. According to HUD officials, factors that could explain the observed differences in the expenses for the public housing and voucher programs of MTW agencies and non-MTW agencies include that MTW agencies typically (1) need more time and resources to develop and implement “innovative” activities, (2) serve hard-to-serve households such as those experiencing homelessness, and (3) provide additional services to the households they serve as a result of the funding flexibilities. According to a University of North Carolina at Chapel Hill study, nearly all MTW agencies have used program flexibility to provide supportive housing for various hard-to-serve populations, including the previously homeless, mentally disabled, developmentally disabled, formerly incarcerated, domestic abuse victims, youth aging out of foster care, and those with substance abuse issues. Some of these programs were provided through sponsor-based voucher programs administered by partner agencies, which required coordination between the MTW agency and the partnering agencies. Limitations in HUD data make it difficult to more fully explain the differences that may affect the number of households served. For instance, HUD cannot measure how participation in the demonstration affected the occupancy and voucher unit utilization rates of MTW agencies. As previously discussed, HUD uses FDS to account for the agencies’ MTW funds, but once combined in the system, the funds are decoupled from the original funding source and it is difficult to determine how these funds were used. As described earlier, although FDS data could be used to illustrate how many agencies transferred voucher funding to their public housing program, these data could not be used to illustrate how many agencies transferred public housing funding to their voucher program because, according to HUD officials, FDS does not identify the source of funding that is available for the voucher program and local, nontraditional activities. Similarly, FDS cannot measure expenses that were for local, nontraditional activities because FDS expenditure categories are not tailored to the MTW demonstration. HUD officials said the reporting of expenses associated with local, nontraditional activities varies by MTW agency, which affects where FDS captures such expenses. HUD has not made changes to FDS because, according to HUD officials, FDS is an accounting system that tracks agencies’ year-end financial activity and, therefore, is not designed to keep track of these data. Furthermore, historical data do not exist on the households served through local, nontraditional activities. Although HUD provided us a spreadsheet it compiled in July 2017 with data on the number of households served through local, nontraditional housing assistance activities from 2009 through 2016, HUD had to manually compile the spreadsheet because its PIC system does not capture data on these households. HUD officials said the agency was considering capturing some data on local, nontraditional households in PIC, but making this change would require HUD and MTW agencies to devote resources to update their systems. HUD previously considered making changes to the system. In 2012, HUD issued a Federal Register notice requesting public comment on changes to the system to track households provided assistance through local, nontraditional activities. According to the notice, agencies had not been reporting these families into the system, which made it difficult to accurately account for the number of MTW families being served. The notice further stated that the MTW Office was manually collecting data on the number of families served each year but the PIC system needed to be revised to make information collection easier for MTW agencies and HUD. HUD officials said HUD did not have the information technology resources needed to make this change in PIC. Federal internal control standards state that management should use quality information to achieve the entity’s objectives. Additionally, one of the statutory objectives of the MTW demonstration is to reduce costs and achieve greater cost-effectiveness in federal housing expenditures, and a key demonstration requirement is to assist substantially the same total number of eligible low-income families under MTW as would have been served absent the demonstration. As discussed previously, intermingled funding streams, the purpose and structure of FDS, and limitations in PIC have combined to limit the data collected and readily available on the MTW demonstration. According to HUD officials, it would be difficult for HUD to require existing agencies to report additional financial data because doing so would require changes to the standard agreement, which generally cannot occur without mutual agreement between the agencies and HUD. Yet agencies’ specific reporting obligations are not set forth in the general standard agreement but rather in Attachment B, which HUD already expanded without requiring an amendment to the standard agreement in 2011 and 2013 and proposed to do in 2016. The standard agreement states that agencies must provide in their annual plan the information required in Attachment B, and under the standard agreement, HUD retains flexibility to determine what constitutes satisfactory completion of the annual plan. Further, the standard agreement, which sets forth general covenants for the demonstration and not specific data points or reporting definitions, specifically acknowledges that HUD must have the “flexibility to design and test various approaches” for housing assistance and that the agencies agree “to cooperate fully with HUD” in the monitoring and evaluation of the MTW demonstration. Under the standard agreement, MTW agencies must provide in their annual report “the information necessary for HUD to assess the Agency’s activities,” without specific detail. As with the annual plan, HUD retains flexibility to determine what data agencies must report. Without more comprehensive data on the uses of MTW demonstration funds and households served through local, nontraditional activities, HUD cannot assess the performance of MTW agencies in relation to public housing occupancy and voucher unit utilization rates and program expenses, which could affect the number of tenants served. MTW agencies have accumulated relatively large reserves of voucher funding. The agencies are able to accumulate more reserves because their voucher funding formula differs from the formula used for the traditional voucher program. HUD allocates voucher funds to non-MTW agencies based on leasing rates and subsidy costs from the prior year. As a result, these agencies have an incentive to expend their voucher funding to keep their budget utilization rate high. However, the voucher formula for MTW agencies, which is outlined in an attachment to each agency’s standard agreement, is generally based on the actual, per-unit costs in the year prior to the agency joining the MTW demonstration. Because the voucher allocation is not tied to prior-year subsidy expenses, MTW agencies do not have the same incentive that non-MTW agencies have to use all their voucher funds in a given year. According to 2016 HUD voucher reserve data, the 39 MTW agencies had almost as much voucher reserves as the 2,166 non-MTW agencies combined. Specifically, as of December 31, 2016, MTW agencies had a total of about $1.11 billion in voucher reserves, whereas the 2,166 non- MTW agencies had slightly higher reserves of $1.13 billion. Similar to our analysis above, we compared the voucher reserves MTW agencies held to the reserves comparable non-MTW agencies held. As figure 11 shows, as of December 31, 2016, the median amount of reserves per household held by MTW agencies was $2,462 compared to $480 for comparable non-MTW agencies (a difference of $1,982 or about 5 times higher). After we completed our analysis, HUD provided updated reserve levels as of June 30, 2017, that showed that MTW agencies’ reserves exceeded non- MTW agencies’ reserves. MTW agencies had a total of about $808 million in reserves while non-MTW agencies had reserves of about $737 million. HUD has performed limited oversight of MTW reserves. For example, before 2016 HUD did not capture data that would help it determine the amount of voucher reserves held by MTW agencies. In January 2012, as part of a new cash management requirement for the voucher program, HUD implemented a process to help transition the accrual of excess funds held at the agency level to HUD-held reserves. According to HUD officials, this process was only partially implemented for MTW agencies at that time because voucher subsidy expenses were comingled with expenses associated with other allowable MTW activities in VMS. In 2016, HUD added new fields in VMS to distinguish various MTW nonvoucher subsidy expenses (such as those for capital improvements of existing public housing units and operation of local, nontraditional activities) from unspent funding. According to HUD officials, these enhancements to VMS now allow HUD to keep track of MTW agencies’ reserves. Consequently, in 2016, HUD started cash reconciliations for MTW agencies, consistent with the cash management procedures for non-MTW agencies. HUD also does not have a process to systematically determine if MTW agencies have public housing reserves. Unlike for the voucher program, HUD was unable to determine the extent to which MTW agencies had unspent public housing funding in reserves. According to HUD officials, FDS tracks overall MTW reserves but HUD cannot distinguish between public housing and voucher reserves because the MTW funds are combined into a single account and because HUD does not have a system similar to VMS that separately tracks public housing reserves for MTW agencies. According to federal internal control standards, management should internally communicate the necessary quality information, such as through written communication, to help achieve the agency’s objectives. Management should design control activities—policies, procedures, techniques, and mechanisms—to achieve objectives and respond to risks. Maintaining comprehensive written policies and procedures will help ensure that control activities are in place to address risks and carry out management directives. We also developed criteria—a set of questions— that agency managers and Congress could use to identify and manage fee revenue instability, including identifying common principles and leading practices for managing reserve funds. For example, managers should ask what level of reserves is to be maintained. In addition, they should consider establishing minimum and maximum reserve levels to ensure accountability and adherence to the reserve’s goals, justifying the numbers with program data and risk management considerations. When established reserve goals have been achieved, such as to fund planned capital investments, the level of reserve should be assessed for reasonableness. However, HUD has not developed and implemented a process to monitor MTW reserves. Specifically, it does not monitor existing MTW agencies’ reserves to determine what agencies plan to do with these reserves and assess whether the plans are reasonable given the amount of reserves. HUD officials said it would require a significant amount of time to individually compare the MTW agencies’ reserves to their planned activities. However, HUD officials said that the draft operations notice for the MTW expansion proposes requiring that expansion agencies hold no more than 1 year of voucher subsidy funds in reserves. But the notice did not outline a plan to evaluate whether this cap was appropriate, and HUD has not yet finalized the notice. Without a process to monitor existing MTW agencies’ plans for their reserves and the appropriateness of the cap for expansion agencies, HUD cannot provide reasonable assurance that MTW agencies have sound plans for expending their reserves. HUD does not have a framework—standard definitions for rent reform and self-sufficiency, clear guidance on reporting requirements, or analysis plans—for monitoring the effect of rent-reform, work-requirement, and time-limit policies. HUD’s definition of rent reform is unclear, leading to agencies inconsistently categorizing some policies and not reporting required information for rent-reform policies. Federal internal control standards state that management should use quality information—relevant and reliable data—to achieve the entity’s objectives. HUD defines rent reform as “any change in the regulations on how rent is calculated for a household.” Under traditional public housing and voucher program rules, an assisted household generally must contribute the greater of 30 percent of its monthly adjusted income or the housing-agency established minimum rent—up to $50—toward its monthly rent. Statute and HUD regulations direct how public housing agencies are to certify tenant income and determine a participating household’s tenant rental payments. Non-MTW agencies must implement this determination process when a household first joins the program and then on a regular basis. In addition, the total housing costs, which are used to calculate a household’s tenant rental payment, include both the rent for the unit and utility costs. As such, an agency is responsible for establishing and maintaining a utility allowance schedule that provides reasonable allowances for tenant-paid utilities. MTW agencies can propose rent- reform policies that make changes to these program rules, such as changing how often tenants are recertified, eliminating certain exclusions or deductions, or changing the approach agencies use to determine a household’s tenant contribution. HUD has 15 categories of activities it considers to be rent reform under the MTW demonstration, but does not further define the activities under each category (see table 4). Based on our review of MTW agencies’ 2015 annual reports, we identified 194 activities that involved one or more rent-reform changes based generally on HUD’s categories of rent-reform activities. When we requested that agencies provide information on their rent- reform activities, several MTW agencies asked for clarification on how rent reform was defined and what activities fell into this category. Based on our analysis of the agencies’ 2015 annual reports, we found five agencies did not consider 15 of the 194 activities we identified to be rent reform using HUD’s definition. Based on our review of the agencies’ 2011–2016 annual plans, we found that some agencies did not report information they are required to report when proposing a rent-reform activity in their annual plans. Based on our review of the 2015 annual reports, we found that 83 of the 194 policies we identified as rent reform did not include any of the hardship data HUD requires agencies to report for rent-reform activities. Officials from some MTW agencies said they did not agree with some of the categories HUD considers to be rent reform. For example, officials from three agencies told us that they did not consider changes to the recertification schedule to be rent reform because such changes do not change how rent is calculated, only the frequency of the calculation. Officials from one agency said that HUD’s definition did not match their agency’s definition because the agency restricts its view of rent reform to any change that affects the actual rent calculation. HUD’s definition includes any change that affects the process related to rent. Officials from another agency told us that they believe HUD does not uniformly apply its definition of rent reform when reviewing agencies’ policies. HUD officials also told us that they plan to clarify the rent-reform definition for expansion agencies. But, as noted previously, HUD told us that making changes for existing MTW agencies could be difficult because doing so could require changes to the standard agreement, which generally cannot occur without mutual agreement between the agencies and HUD. However, HUD’s definition for rent reform is set forth in Attachment B, which HUD already has revised without changes to the standard agreement and is currently revising to clarify existing reporting requirements. Without a more clear definition of rent reform and specific criteria or standards with which to classify activities as rent reform, HUD lacks the quality information needed to monitor all rent-reform activities. Although one of the requirements of the MTW demonstration is to establish a reasonable rent policy to encourage employment and self- sufficiency, HUD has not defined self-sufficiency, but rather allowed each agency to develop its own definition. To measure the extent to which certain MTW activities, including rent-reform activities, encourage households to achieve self-sufficiency, HUD requires MTW agencies to report on the number of households that transitioned to self-sufficiency, among other things. According to Attachment B of the standard agreement, MTW agencies are allowed to define self-sufficiency for each activity that is tied to this HUD metric. MTW agencies’ definitions of self-sufficiency can diverge widely and sometimes are inconsistent within an MTW agency. Some examples include defining self-sufficiency as attaining a total gross household income at 80 percent of the area’s paying a minimum rent of $225; voluntarily terminating housing assistance and other forms of government assistance; and attaining a household income of 50 percent of the area median income, even if the family may be receiving other state benefits. In addition, some agencies use multiple definitions of self-sufficiency. For example, one agency uses three definitions for self-sufficiency (one for its public housing minimum rent activity, one for its voucher rent-reform activity that combined various changes, and another for its public housing earned income disregard alternative activity). Previously, we found that clarity, reliability, and balance are three of several key attributes of successful performance measures, which are means of objectively assessing the outcomes of programs, products, projects, or services. A measure has clarity when it is clearly stated and the name and definition are consistent with the methodology used for calculating the measure. A measure that is not clearly stated can confuse users and cause managers or other stakeholders to think performance was better or worse than it actually was. A measure is reliable when it produces the same result under similar conditions. Lack of reliability causes reported performance data to be inconsistent and adds uncertainty. Another key attribute of successful performance measures is balance, which exists when measures ensure that an agency’s various priorities are covered. Performance measurement efforts that overemphasize one or two priorities at the expense of others may skew the agency’s performance and keep managers from understanding the effectiveness of their program. According to HUD officials, they have not defined self-sufficiency for MTW agencies because they want to give agencies the ability to address local needs. However, the individualized definitions have led to measurements of self-sufficiency that cannot be consistently evaluated across activities or agencies. In addition, officials said that it would be inappropriate for them to develop a definition of self-sufficiency for the MTW demonstration because HUD has not defined it for the department. However, despite the lack of an agency-wide definition of self-sufficiency, HUD regulations define self-sufficiency for certain other HUD programs. As such, HUD also could develop a self-sufficiency definition for the MTW demonstration. Without a more standardized definition of self-sufficiency for the MTW demonstration, HUD cannot collect consistent information that would allow for the evaluation of the effect of MTW rent-reform and occupancy policies on tenants. HUD’s guidance on how agencies are to perform impact analyses, reevaluate activities, and establish hardship policies has not described the elements of the analysis, required submission of reevaluations, or described elements of hardship policies. Attachment B of agencies’ standard agreement contains general instructions for reporting information in MTW annual plans and annual reports, including on rent- reform activities. For example, when an agency proposes a rent-reform activity, the agency must conduct an impact analysis, describe how it will annually reevaluate the activity, and develop a hardship policy for the activity. According to HUD officials, HUD implemented these reporting requirements for rent-reform activities because they could have significant effects on tenants. Attachment B suggests agencies take four steps when developing an impact analysis and include the results, including describing the rent- reform activity and identifying the intended and possible unintended effects of the activity; however, it does not provide any explanation or suggestions for how agencies should approach each step. According to HUD officials, these steps are not required and the only other guidance provided to agencies to monitor the effect of rent-reform activities is draft guidance from 2009. The 2009 draft guidance reiterates the four suggested steps of an impact analysis and provides a narrative explanation of the purpose of each step along with examples; however, agencies are not required to follow the guidance and HUD never finalized it. We reviewed the impact analyses agencies reported in their annual plans from 2011 through 2016 and found that agencies’ impact analyses for their rent-reform policies varied widely in the type of information included and level of detail. For example, a majority of impact analyses included whether the activity would increase or decrease tenants’ rent burden and a majority included other benefits or costs to tenants, but analyses less often discussed possible unintended consequences of their rent-reform policies. In addition, some agencies did not include the same type of information across the analyses of their activities. One agency provided an example of how a hypothetical tenant’s rent could change when the agency moved to biennial recertifications, but did not analyze how tenants’ rent could change for its minimum rent or tiered rent policies. Another agency included the potential impact on the agency for each of its proposed activities, but only analyzed the potential rent burden on tenants for one activity. In addition, the level of detail included in the impact analyses varied. For example, in discussing a policy that would change what sources of income were included in a tenant’s rent calculation, one agency’s impact analysis stated that the change would save money for tenants. An impact analysis for a similar policy from another agency included the number of tenants who would be affected by the policy and a dollar estimate of how much money tenants could save. Activities that might be considered administrative, such as changes to the frequency of tenant recertifications, were less likely to include details such as analysis of the rent burden on tenants than were other activities. In several agencies’ impact analyses, as well as in interviews with agency officials, agencies generally indicated that they think of these MTW policies or activities as being good for tenants, which may explain why agencies were less likely to discuss burden on tenants. HUD officials acknowledged the need for more detailed guidance and said they planned to provide such guidance for the expansion agencies. HUD officials said that they have not created such guidance for the existing agencies because they have been focused on the recent expansion of the demonstration and because doing so could require changes to the standard agreement. However, the steps for an impact analysis are contained in Attachment B, to which, under the standard agreement, agencies must adhere to satisfy their annual reporting obligations. Further, HUD has already revised Attachment B and agencies’ reporting requirements contained therein on multiple occasions without requiring changes to the standard agreement. Officials stated they could encourage existing agencies to follow the guidance for the expansion agencies. Federal internal control standards state that management should externally communicate the necessary quality information to achieve the agency’s objectives. By framing the steps in Attachment B as suggestions and not prescribing the elements of impact analyses, HUD cannot consistently collect the type of information it needs to assess the effect of MTW activities on tenants across agencies. For example, according to HUD officials, one of the purposes of the impact analysis is to encourage agencies to consider potential unintended consequences of their activities. However, unintended consequences cannot be assessed without more detailed impact analyses. Attachment B does not describe the elements MTW agencies must include in their annual reevaluation, and HUD does not require MTW agencies to submit the results of those reevaluations. According to Attachment B, when agencies propose a rent-reform activity in their annual plan, they should provide an overview of how they will annually reevaluate the proposed activity and revise the activity as necessary to mitigate the negative effects of any unintended consequences. However, it does not provide any further detail or examples of what agencies should annually reevaluate. In addition, while HUD requires agencies to perform annual reevaluations of rent-reform activities, HUD guidance does not require MTW agencies to report the results of their annual reevaluations. According to federal internal control standards, management should externally communicate the necessary quality information to achieve the agency’s objectives. Based on our review of agencies’ annual plans submitted from 2011 through 2016, about one-third of the rent-reform policies proposed by agencies included a description of how agencies planned to annually reevaluate the policies. The remaining proposals either did not include a description or agencies stated that they would evaluate the activity annually without providing further description of how they would perform the evaluation. When we requested that agencies provide their 2015 annual reevaluations of their rent-reform policies, several of the MTW agencies were confused about what we meant by annual reevaluation. Some of those agencies asked if we were referring to their annual report and one agency asked how an annual reevaluation was different from an impact analysis. When we received documentation of what the agencies considered to be the annual reevaluations of their rent-reform activities, 30 of the agencies provided us information they are required to include for all of their activities in their annual reports. For example, agencies must include a description of their activities and their impact, compare policy outcomes to HUD metrics, and explain challenges they faced if benchmarks were not achieved. Most agencies referred us to all or part of this information. However, some agencies provided analyses that went beyond those required for annual reports, including evaluations from third-party researchers. For example, one agency partners with a local university to conduct an annual survey that allows the agency to assess the effect of its rent-reform activities on households. During the course of our work, a HUD official said the agency had not required MTW agencies to report annual reevaluations because, as long as agencies had a plan to annually reevaluate their activities and HUD had the ability to request the reevaluations if concerns arose, HUD did not want to require agencies to report information HUD did not intend to analyze. HUD officials later stated that the agency plans to provide more detailed guidance for the expansion agencies and has been updating Attachment B to clarify that agencies’ annual reports must include the results of their annual reevaluations of their rent-reform activities. In addition, HUD officials said they could issue guidance that encouraged existing agencies to follow the guidance for the expansion agencies but it would be difficult to require existing agencies to include specific elements in these annual reevaluations without changes to the standard agreement. However, the standard agreement merely requires that MTW agencies fulfill the annual reporting requirements set forth in Attachment B, which provides the detailed description of the required elements of the annual plan and report and which HUD has already revised on multiple occasions without requiring changes to the standard agreement. Because HUD allows agencies to determine the process for reevaluating their activities, most MTW agencies have not collected or reported additional information on rent-reform activities (including effects or unintended consequences) outside of the requirements of their annual reports. This leaves HUD and the agencies themselves less able to assess the effects of MTW activities on tenants. While MTW agencies must establish a hardship policy to define the circumstances under which households may be exempted or receive temporary waivers from a new rent-reform activity, Attachment B does not define what elements must be included in the hardship policy. The nonbinding draft guidance from 2009 we previously discussed suggested four questions hardship policies should address (including the process households would use to request an exemption or waiver and how hardship cases would be resolved). Officials from the seven agencies we interviewed said they looked to a range of tools to create their hardship policies. For example, officials from one agency said they relied on the 2009 draft guidance and officials from another agency said they relied on Attachment B when developing their policies. Officials from three other agencies said they reviewed the hardship policies of other MTW agencies, had conversations with HUD while planning the activity or waiting for HUD’s review of their annual plan, or looked to relevant federal regulations. In contrast, officials from another agency said that there was no guidance available on how to create their hardship policies because their agency joined the demonstration the year it began. Our review of MTW agencies’ hardship policies for rent-reform activities showed that while these hardship policies had some commonalities, they also were inconsistent in terms of the type of information included. For example, of the 84 hardship policies we reviewed, MTW agencies included a discussion of how the agency processes a hardship complaint in 56 policies and what remedies are available for residents approved for a hardship exemption or waiver in 75 policies. In contrast, 26 policies included information about whether tenants have the ability to reapply for a hardship exemption or waiver, and 26 policies mentioned if the agencies have different rules for the elderly or persons with disabilities. In addition, although most hardship policies generally discussed how a tenant may claim a hardship and apply for an exemption, some agencies were much more specific about the process. For example, one agency stated only that tenants may request a hardship exemption in writing, while another agency explained which application a tenant needed to fill out, what supporting documentation to include, and how to submit the application. Some agencies have created more parameters around a tenant’s ability to request a hardship exemption or waiver than others. For example, some hardship policies are time-limited (that is, tenants have a certain window of time in which to apply). One agency instituted a hardship policy for its minimum rent that stated that tenants had 15 days from receipt of notice of their new household tenant rental payment to apply for a hardship exemption or waiver. Another agency instituting a hardship policy for a similar activity did not seem to impose a time limit for a tenant to request an exemption. In addition, some hardship policies provided relief for current tenants. For example, one-third of agencies created a hardship policy for at least one of their activities that either exempted current residents from the rent-reform activity or provided some form of temporary relief as the rent-reform policy was implemented. We also found variation in the information MTW agencies were able to provide on the households that requested a hardship exemption. We asked all the MTW agencies to provide us a list of all tenants who requested a hardship exemption in 2011–2015, including the result of each request (denied or approved), the current status of each tenant, and the reason the tenant was no longer receiving housing assistance, if applicable. Of all the MTW agencies, five said they had not received any requests for hardship exemptions. Three agencies were only able to provide us information on those hardship requests that were approved, two agencies did not indicate if the requests they received were approved or denied, and one agency did not provide any data because it could not distinguish hardship requests for its traditional programs from its MTW activities. Additionally, five agencies did not provide the reasons why tenants who requested a hardship exemption were no longer receiving assistance. The remaining 22 agencies were able to provide the information as requested. Tenants and advocates expressed mixed opinions about the rent-reform hardship policies created by the MTW agencies we interviewed. Some tenants with whom we spoke said they were aware of rent-reform hardship policies the agencies developed. For example, tenants who participated in one of our group meetings told us that during their income recertification the case worker assigned to their case provided them a checklist that outlined each of the agency’s hardship policies. When we spoke with advocates who work with tenants subject to MTW activities, some said most tenants do not know about the hardship policies available to them. Some tenants and advocates with whom we spoke said the process for requesting a hardship could be difficult. For example, one tenant said that although the MTW agency mailed tenants “frequently asked questions” that described the hardship policy, the document was confusing and included a citation to the Federal Register for more information, which was difficult for tenants to access. Advocates at one organization also said tenants asked for help because the tenants applied for a hardship waiver through their case manager, but never received a response. In contrast, during these meetings some other tenants told us that they had no issues with the hardship policies or the way in which the MTW agencies implemented them. As discussed previously, federal internal control standards require agencies to communicate effectively with external stakeholders to help achieve agency goals. While HUD’s proposed update to Attachment B provides more detail than the current version, HUD officials said it could be difficult to develop more descriptive guidance for existing MTW agencies because doing so could require changes to the standard agreement. In addition, officials said they had not been able to develop more guidance for existing agencies because of their focus on the expansion demonstration. However, the standard agreement merely requires that MTW agencies fulfill the requirements contained in Attachment B, which HUD has already revised on multiple occasions without requiring changes to the standard agreement. Officials said that they plan to provide more descriptive guidance for expansion agencies and encourage existing agencies to follow such guidance. By not providing more specific direction to the MTW agencies about what to include in their hardship policies and therefore what is communicated to tenants, existing agencies may not be adequately communicating all of the information tenants need to understand the circumstances in which they may be exempted from rent-reform activities. HUD requirements for MTW agencies that establish policies for work requirements and time limits are largely inconsistent with requirements pertaining to rent-reform activities (see table 5). Although HUD has said it considers work-requirement and time-limit activities to have a great and direct impact on tenants, the current MTW agencies in the demonstration are not subject to the same reporting requirements when proposing those policies as when proposing rent-reform activities. For example, as previously discussed, HUD guidance in Attachment B requires agencies to include an impact analysis, annual reevaluation, and hardship policy for rent-reform activities in their annual plans when the activity is proposed. However, Attachment B does not include similar requirements for proposed work-requirement or time-limit policies. Further inconsistencies include that Attachment C of the standard agreement, which lists the various MTW flexibilities available to agencies, requires MTW agencies to create a hardship policy if they establish a time-limit policy for public housing assistance. However, HUD did not develop guidance requiring agencies to report on their hardship policies for time-limit policies for public housing assistance. Furthermore, HUD does not have a similar requirement for time-limit policies established for voucher assistance. In addition, in the Federal Register operations notice for the expansion of the MTW demonstration published in January 2017, HUD proposed requiring the new MTW agencies to conduct an impact analysis and develop a hardship policy for rent-reform and time-limit policies, but develop only a hardship policy for work requirements. As previously discussed, federal internal control standards require management to design control activities—policies, procedures, techniques, and mechanisms—in response to the entity’s risks. In determining the necessary level of precision for a control activity, management is to evaluate, among other things, consistency of performance. A control activity that is performed routinely and consistently generally is more precise than one performed sporadically. HUD officials have said that they consider rent-reform, work-requirement, and time-limit policies to have a great and direct impact on tenants. HUD was not able to provide an explanation as to why they do not require similar reporting for all of these activities. HUD officials said they did not know why MTW agencies were not initially required to report on impact analyses, annual reevaluations, and hardship policies associated with work-requirement and time-limit policies in general. However, they said, currently, these policies are typically implemented in conjunction with a rent-reform activity so there is still reporting on the combined policies. HUD officials also stated that if an agency proposed an activity with a time limit for public housing, the MTW coordinator reviewing the agency’s annual plan would ensure that a hardship policy was in place. In addition, when MTW staff review a proposed work requirement for both the public housing and voucher programs and a proposed time limit for the voucher program, staff suggest that MTW agencies adopt hardship policies and conduct impact analyses for these policies. HUD officials also stated that the agency plans to require expansion agencies to develop an impact analysis, annual reevaluation, and hardship policy for rent-reform, work-requirement, and time-limit policies. Although HUD officials said it would be difficult to set a similar requirement for existing MTW agencies because doing so would require changes to the standard agreement, they stated they could update Attachment B to incorporate the requirement for a hardship policy for public housing time limits and develop guidance encouraging existing agencies to comply with the additional requirements put in place for the expansion agencies. Without taking these steps, HUD will miss an opportunity to collect information needed to evaluate the effect of work- requirement and time-limit policies on tenants. Although HUD requires MTW agencies to report annually on their rent- reform, work-requirement, and time-limit policies, HUD could not provide us with documentation of how it analyzed, used, or planned to use the information it received from agencies on a continuous basis. According to HUD officials, because of the recently resolved backlog of annual reports, the MTW Office now can begin to use the years of reported data it previously had not used. Officials added they provide the annual plans and reports to other departments in HUD to conduct ad hoc analysis and that other HUD offices have used MTW plans and reports when proposing new rules or legislation related to housing. For example, officials said HUD used MTW plans and reports when working on HUD’s 2016 rule intended to provide greater flexibility for agencies administering HUD’s rental assistance programs. HUD provided us documentation showing that it used lessons learned from the MTW demonstration to inform legislative proposals in the agency’s fiscal year 2018 and 2019 budgets. Also, MTW officials said they intend to use the data in annual reports to inform some oversight rules. When asked about the agency’s plan to analyze the information provided in the annual plans and reports, HUD officials said it had awarded a contract to the Urban Institute to perform a retrospective evaluation of the demonstration, and the results will be available in 2018. Officials said although they have not finalized their reporting requirements for agencies in the expansion, these agencies likely will not be required to create annual plans or reports but instead to annually create a supplemental document to their annual public housing plan. With those agencies, HUD will be able to learn from each of the cohorts about the effect of a specific policy being evaluated. However, the plan to analyze the supplemental documentation and cohorts of the expansion agencies does not address how HUD plans to use the information it receives from the current MTW agencies. Federal internal control standards state that management should establish monitoring activities and evaluate results. Analysis (evaluation of results) contributes to the operating effectiveness of monitoring. The internal control standards also state that management should use quality information to achieve the entity’s objectives. In doing this, management is expected to use quality information to make informed decisions and evaluate the entity’s performance in achieving key objectives and addressing risks. Because the MTW Office has not systematically analyzed or evaluated the information it requires MTW agencies to report—or determined how best to evaluate it—the agency cannot assess the effect of MTW rent- reform, work-requirement, and time-limit policies on tenants. More specifically, without a plan for analyzing information in agencies’ impact analyses, annual reevaluations, and hardship policies, HUD cannot monitor the effect of rent-reform, work requirement, and time limit policies on tenants. These limitations also extend to the definitional and guidance issues we previously discussed. As a result, without a comprehensive framework—standard definitions, clear guidance on reporting requirements, and analysis plans—HUD cannot provide assurance that it is adequately monitoring how MTW activities affect tenants. The MTW demonstration is on the brink of significant expansion, but HUD does not yet have the people, data, and processes in place to effectively oversee agency participants and assess the demonstration’s performance and effects on tenants. Workforce planning. Insufficient staffing for the MTW demonstration already has had negative effects. For instance, HUD has not always reviewed annual reports that include information needed to determine the demonstration’s effect on tenants in a timely manner, annually assessed whether current MTW agencies comply with demonstration requirements, and fully documented its review processes. When complete, expansion of the demonstration would more than triple the number of MTW agencies. By finalizing its workforce planning (including an assessment of competencies and skills needed) and documenting its compliance review process, HUD can provide assurance that it would be positioned to oversee an expanded demonstration before new agencies start being added in 2018. Data collection. Our comparison of public housing occupancy and voucher unit utilization rates and program expenses among MTW and non-MTW agencies raises questions about agency performance and use of funding that cannot be fully answered with current data. The differences among agencies may result in part from the MTW demonstration’s funding flexibilities. However, HUD is limited in its ability to readily determine the extent to which MTW funds were used for other allowable purposes. More comprehensively capturing and tracking data on uses of funding and the characteristics of households served by local nontraditional activities would allow HUD to better assess agency performance. HUD also would be better able to account for differences in outcomes—especially in relation to occupancy and voucher utilization rates and program expenses—that affect the number of tenants served. MTW reserves. The accumulation of relatively large reserves by MTW agencies also raises questions about funding uses. HUD has performed limited oversight of MTW voucher reserves and its data and financial reporting systems are not structured to effectively track public housing reserves. Developing and implementing a process to monitor MTW reserves could help HUD provide reasonable assurance that MTW agencies have sound plans for expending reserves. Framework for assessing effect of rent-reform, work-requirement, and time-limit policies on tenants. The effectiveness of certain MTW activities and their effects on tenants remain largely unknown because HUD does not have a framework—standard definitions for key terms, clear guidance on reporting requirements, and analysis plans—for monitoring rent-reform, work-requirement, and time-limit policies. For example, the variations in reporting on rent reform and self-sufficiency as a result of inconsistent definitions of these terms; limited guidance (often couched as suggestions) HUD provided to agencies for developing impact analyses, annual reevaluations, and tenant hardship policies; and inconsistent treatment of rent-reform and work-requirement and time-limit policies suggest that HUD may have emphasized flexibility to the detriment of oversight. In addition, HUD does not have a plan for assessing the information agencies report on the effect of these policies. Developing such a framework will help both HUD and MTW agencies to assess performance and determine if activities have advanced demonstration goals. We recognize the challenges involved with monitoring the MTW demonstration, but maintain it is important for HUD to take steps to achieve and sustain a better balance between flexibility and prudent oversight. Improving oversight of the demonstration would help HUD assess what MTW agencies have done, including their use of funding. Such information also would help inform Congress and the public about how demonstration innovations have affected tenants. We are making the following 11 recommendations to HUD: The Assistant Secretary for PIH should complete workforce planning for the MTW demonstration to help ensure that PIH has sufficient staff with appropriate skills and competencies to manage an expanded demonstration, including reviewing reports and carrying out compliance reviews in a timely manner. (Recommendation 1) The Assistant Secretary for PIH should more fully document the process for annually assessing compliance with the five demonstration requirements. (Recommendation 2) The Assistant Secretary for PIH should develop and implement a process to track how MTW demonstration funds are being used for other allowable activities, including local, nontraditional activities. (Recommendation 3) The Assistant Secretary for PIH should identify and implement changes to PIC to capture household data for households served through local, nontraditional activities. (Recommendation 4) The Assistant Secretary for PIH should develop and implement a process to monitor MTW agencies’ reserves. (Recommendation 5) The Assistant Secretary for PIH should clarify HUD’s rent-reform definition for the MTW demonstration as part of a framework for monitoring the effect of rent-reform, work-requirement, and time-limit policies on tenants. (Recommendation 6) The Assistant Secretary for PIH should set parameters for HUD’s definition of self-sufficiency for the demonstration, either by providing one definition or a range of options from which agencies could choose, as part of a framework for monitoring the effect of rent-reform, work- requirement, and time-limit policies on tenants. (Recommendation 7) The Assistant Secretary for PIH should revise HUD’s guidance to MTW agencies to make it clear which elements are required in impact analyses, annual reevaluations, and hardship policies and the information required for each element as part of a framework for monitoring the effect of rent-reform, work-requirement, and time-limit policies on tenants. (Recommendation 8) The Assistant Secretary for PIH should develop written guidance for existing MTW agencies that requires a hardship policy for public housing time limits and encourages an impact analysis, annual reevaluation, and hardship policy for work-requirement and time-limit policies for public housing and voucher programs as part of a framework for monitoring the effect of these policies on tenants. (Recommendation 9) The Assistant Secretary for PIH should require an impact analysis, annual reevaluation, and hardship policy for work-requirement and time- limit policies new MTW agencies adopt for their public housing and voucher programs as part of a framework for monitoring the effect of these policies on tenants. (Recommendation 10) The Assistant Secretary for PIH should develop and implement a plan for analyzing the information that agencies report on the effect of rent- reform, work-requirement, and time-limit policies on tenants as part of a framework for monitoring the effect of these policies on tenants. (Recommendation 11) We provided a draft of this report to HUD for comment. In written comments, which are summarized below and reproduced in appendix III, HUD disagreed with three of our recommendations and generally agreed with the remaining eight. In its general comments, HUD made the following points: HUD noted that our report did not identify any harmful effects on tenants as a result of MTW flexibilities. As discussed in the draft report, due to data limitations, we could not evaluate the effect of MTW flexibilities on tenants. Instead, we focused on the extent to which HUD monitored the effects of rent-reform, work-requirement, and time-limit policies on tenants. Furthermore, our analysis of available data showed that MTW agencies had lower public housing occupancy rates and voucher unit utilization rates and higher program expenses than comparable non- MTW agencies, which could affect the number of tenants served. HUD also stated that it seemed we reviewed MTW agencies through the lens of the traditional housing and voucher programs. HUD noted fundamental differences in MTW and non-MTW agency operations and stated it must consider the extensive MTW flexibilities and the locally- designed nature of each MTW agency’s program in administering the demonstration. HUD stated it did not agree with three of our recommendations (discussed below) that it noted would restrict an MTW agency’s ability to exercise MTW flexibility and respond to variations in local markets. As stated in the draft report, we recognize the challenges involved with monitoring the MTW demonstration, but maintain it is important for HUD to take steps to achieve and sustain a better balance between flexibility and prudent oversight. Furthermore, given that the demonstration’s ultimate goal is to identify successful approaches that can be applied to public housing agencies nationwide, we believe we looked objectively and with the appropriate rigor and contextual sophistication at MTW agencies. HUD disagreed with the draft report’s third recommendation to develop and implement a process to track how public housing and voucher funding is being used for other allowable activities, including local, nontraditional activities. HUD stated that funding fungibility and policy flexibility are the core tenets of the MTW demonstration. As a result, identifying and tracking expenses paid from a specific funding source are not necessary and should not be a requirement. We acknowledge the demonstration’s funding and policy flexibility and did not intend for our recommendation to be interpreted solely as a suggestion to track funding sources. We therefore clarified our recommendation to focus on tracking how MTW demonstration funds are being used for allowable activities, such as local, nontraditional activities. HUD stated that the revised HUD Form 50900 or Attachment B (expected to be published in early 2018) would require existing MTW agencies to estimate the cost of each planned activity. Although this would provide some cost information, it would be limited to planned activities only and would not capture actual costs. Therefore, we continue to believe that more comprehensively tracking data on uses of funding would allow HUD to better account for differences in outcomes—especially in relation to occupancy and voucher utilization rates and program expenses—that affect the number of tenants served. HUD disagreed with the fifth recommendation to develop and implement a process to monitor MTW agencies' reserves. HUD stated that there is no language in the 1996 Act that limits the reserves of MTW agencies to a certain level. Although our draft report noted that leading practices for managing reserve funds include considering establishing a maximum reserve level, we did not recommend that HUD set such a reserve level for MTW agencies because we recognized the demonstration’s funding flexibilities. Rather, we recommended that HUD develop a process to monitor MTW agencies’ plans for reserves. HUD also commented that by reviewing and granting approval for all MTW activities that the existing 39 agencies implemented, it already had a process to determine if spending of reserve funds was reasonable. However, as HUD noted in its comments on the draft report’s third recommendation, the agency does not currently require MTW agencies to include the cost of a planned activity when proposing the activity. An approval process that does not include a review of information on planned costs, including the extent to which reserves would be used to fund the activity, is not sufficient because HUD lacks data needed to determine that reserve expenditures are reasonable. Finally, HUD noted that PIH’s Financial Management Division currently tracks the public housing and voucher reserves of MTW agencies. However, this does not address our concern that HUD does not monitor existing MTW agencies’ plans for their reserves and whether the plans are reasonable given the amount of reserves. In order to provide reasonable assurance that MTW agencies have sound plans for expanding their reserves, HUD still would have to develop a process to monitor MTW agencies’ reserves. Therefore, we maintain our recommendation. Similarly, HUD disagreed with our seventh recommendation to set parameters for its definition of self-sufficiency for the demonstration, either by providing one definition or a range of options from which agencies could choose. It noted that the MTW demonstration provides agencies with the ability to develop creative solutions to address local conditions, and a one-size-fits-all approach is not appropriate. HUD stated it intentionally has not developed a standard definition for self- sufficiency, because the definition could depend on local conditions such as employment opportunities and availability of supportive services. We recognized the need for flexibility in our recommendation by suggesting that HUD could develop a range of definitions from which MTW agencies could choose. This approach would provide the necessary flexibility while still allowing HUD to collect the consistent information needed to evaluate the effect of MTW rent-reform and occupancy policies on tenants. Therefore, we maintain our recommendation. HUD generally agreed with our remaining eight recommendations. For example, HUD agreed with the draft report’s first recommendation on workforce planning, but requested that due to the cross-cutting nature of MTW, we expand the recommendation to include other PIH offices. We acknowledge that the staff needed to manage the expanded demonstration may be found outside the MTW Office, and therefore we modified our recommendation. HUD also agreed with the second recommendation to more fully document the process for annually assessing compliance with the five demonstration requirements and said it will finalize internal written procedures in early 2018. In addition, in commenting on the fourth recommendation, HUD described plans to update its data system to capture information on households served through local, nontraditional MTW activities. Furthermore, in regard to the eighth recommendation, HUD noted that it plans to develop guidance for MTW agencies for the monitoring of high-impact activities such as rent reform, work requirements, and time limits. Finally, in commenting on the eleventh recommendation, HUD stated it will improve its process of analyzing the data MTW agencies provide on high-impact activities. In commenting on our workforce planning finding, HUD made the following points: HUD stated that our finding that planning for the MTW expansion workforce structure has not been completed is not an accurate characterization. It noted that HUD completed a workforce analysis and hired five additional staff in 2016 in anticipation of the MTW expansion. In our draft report, we acknowledged steps that HUD took to increase the staffing levels of the MTW Office. However, we found that in its workforce analysis, HUD had not assessed the knowledge, skills, and abilities needed to implement an oversight structure for the MTW expansion demonstration. HUD acknowledged in its response to the recommendation that its workforce planning efforts will continue in 2018. HUD said our draft report did not discuss two other factors (beyond insufficient staff) that affected oversight of the MTW demonstration: (1) 2013 was the first year HUD assessed each agency’s compliance with the five demonstration requirements, and (2) from 2013 to 2015, HUD was in protracted and complex negotiations with the existing MTW agencies to determine the terms of the extension of their MTW participation. Our draft report acknowledged both factors. Specifically, we noted that HUD developed a process for assessing compliance with the five demonstration requirements in response to a recommendation in our 2012 report and that the process was implemented in 2013. Our draft report also stated that HUD officials noted that in 2014 and 2015 existing staff in the MTW Office had to focus on other priorities, including renegotiating the standard agreement, and then in 2016 on implementing the expansion of the demonstration. HUD said that even with limited staff, MTW agency plans had been reviewed and approved within the required time frames. In commenting on our data collection finding, HUD made the following points: Related to our multivariate statistical analysis to examine any association between MTW flexibilities and program outcomes, HUD stated that HUD and MTW agencies historically found it difficult to establish comparison groups because MTW and non-MTW agencies implement significantly different interventions. We agree that comparisons of MTW and non- MTW agencies are difficult to make. We acknowledge that MTW agencies differ substantially from non-MTW agencies on factors such as size and market housing costs. Accordingly, we used statistical techniques to improve on simple comparisons between MTW and non- MTW agencies. These techniques enabled us to identify a group of comparison non-MTW agencies that were similar to MTW agencies on important factors such as geographic location, households served, and county median rents. We then compared outcomes between the two groups of agencies over a number of years (2009 through 2015). We did not compare a single MTW agency to a non-MTW comparison group, as HUD stated. For more detailed information on our analysis, see appendix II. HUD also stated that our finding that MTW agencies had higher tenant services expenses for the voucher program than non-MTW agencies was an expected outcome (because the demonstration encourages MTW agencies to engage in employment, self-sufficiency programming, and tenant services). In our draft report, we stated that the results of the analysis were consistent with MTW agencies having more flexibility to use funds to provide tenant services. Furthermore, HUD said that a comparison of voucher administrative expenses for MTW and non-MTW agencies was skewed and not a valid comparison because administrative expenses for MTW agencies included voucher administrative expenses and other administrative expenses not permitted under the traditional voucher program. Differences in financial and performance outcomes that only MTW flexibilities allow, such as a broader range of administrative expenses, represent the potential effects of the demonstration, not a source of bias. The purpose of our analysis was to determine any association between MTW flexibilities and program outcomes. Because MTW rules allow for additional administrative expenses, it was appropriate to include these expenses in our analysis. In addition, HUD stated that that it had requested the list of the comparison group of non-MTW agencies to MTW agencies and suggested the list be included in our report. The agency noted that without this information, HUD was not able to validate our analysis. As noted previously, our analysis was not a simple comparison of MTW and non-MTW agencies. We developed a comparison group, applied algorithms based on certain assumptions, and conducted sensitivity analyses that tested these assumptions. Therefore, simply providing the list would not enable HUD to reproduce our analysis. Furthermore, we selected the variables for matching because they were similar across all agencies in each group (that is, the full distributions), not for any particular pair of matched agencies. Consequently, we evaluated the quality of our comparison group using the distributions of these variables across all agencies in each group. We included those statistics in our report, rather than the identity of particular agencies, to encourage systematic evaluations of the matched comparison agencies using aggregate statistics, rather than anecdotal evaluations of particular matched pairs. Finally, we communicated with HUD throughout the review about our data analysis. For example, we met with HUD to discuss our methodology, provided initial results, and worked with HUD officials to ensure we were using appropriate data fields. HUD also provided technical comments, which we incorporated as appropriate. We considered one comment to be more than technical in nature. Specifically, in response to our finding that HUD does not require MTW agencies to submit the results of their annual reevaluations of the impact of rent-reform activities, HUD officials stated that they consider the annual report (and information therein) to be the annual reevaluation of rent-reform activities. However, Attachment B does not include a requirement that agencies report the results of their annual reevaluations. Furthermore, if the information currently required to be included in the annual report satisfied the annual reevaluation requirement, then there would be no need for HUD to update Attachment B to clarify that agencies’ annual reports must include the results of their annual reevaluations, as the agency plans to do. Therefore, we maintain our finding and made revisions to the report to clarify what is currently required in Attachment B. As agreed with your office, unless you publicly announce the contents of this report earlier, we plan no further distribution until 30 days from the report date. At that time, we will send copies of this report to the appropriate congressional committees, the Secretary of the Department of Housing and Urban Development, and other interested parties. In addition, the report will be available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-8678 or garciadiazd@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix IV. Our objectives were to examine (1) the Department of Housing and Urban Development’s (HUD) oversight of agencies participating in the Moving to Work (MTW) demonstration, including agency reporting and compliance with demonstration requirements; (2) any association between MTW flexibilities and program outcomes, including public housing occupancy rates and voucher unit utilization rates; and (3) the extent to which HUD monitored effects of rent-reform, work-requirement, and time-limit policies on tenants. For all our objectives, we interviewed officials from the following seven MTW agencies: Boulder Housing Partners (Boulder, Colorado); Chicago Housing Authority (Chicago, Illinois); Delaware State Housing Authority (Dover, Delaware); Lincoln Housing Authority (Lincoln, Nebraska); Louisville Metropolitan Housing Authority (Louisville, Kentucky); Housing Authority of the County of San Bernardino (San Bernardino, San Diego Housing Commission (San Diego, California). In selecting these agencies, we focused on agencies that had implemented major rent-reform changes and work-requirement and time- limit policies based on information in a study conducted in January 2015 by the Center for Urban and Regional Studies at the University of North Carolina at Chapel Hill. We focused on these policies because, according to HUD, they have a great and direct impact on tenants. We also considered agency size, length of time in the demonstration, and geographic diversity. Although the results of the interviews cannot be generalized to all MTW agencies, they provide insight into the ways in which agencies implemented MTW flexibilities and report to HUD, among other things. In addition, we interviewed representatives of the following research groups to discuss their recent or ongoing work on the MTW demonstration: Abt Associates, the Center for Urban and Regional Studies at the University of North Carolina at Chapel Hill, HAI Group, Public and Affordable Housing Research Corporation, and the Urban Institute. We also interviewed representatives of affordable housing advocacy groups such as the Council of Large Public Housing Agencies; National Association of Housing and Redevelopment Officials; National Leased Housing Association; and Public Housing Authorities Directors Association. Finally, we interviewed resident advocacy organizations such as the Center on Budget Policy and Priorities, National Housing Law Project, and National Low-Income Housing Coalition. To select the groups to interview, we reviewed our 2012 report on MTW, identified organizations through our background literature review, and obtained recommendations from those we interviewed. To examine HUD’s oversight of MTW agencies, we reviewed our 2012 report, relevant HUD policies and procedures, and HUD documentation relating to compliance with the demonstration. Specifically, we reviewed the standard agreement that governs the participation of the existing 39 MTW agencies in the demonstration and HUD’s guidance on agency reporting and the five demonstration requirements. We also interviewed HUD officials about the processes HUD uses to review the agencies’ annual reports and assess compliance with the demonstration requirements. We also reviewed workforce analyses and interviewed HUD officials about their resource needs and plans to monitor the current MTW agencies and any agencies that may join the MTW demonstration through its expansion. We compared relevant internal control standards that apply to federal agencies and best practices we identified for workforce planning with HUD’s monitoring policies and procedures. To assess the extent to which HUD follows its processes, we reviewed HUD’s documentation of compliance assessments from 2013 through 2016, the only years for which HUD had completed such analysis. To identify and examine any association between MTW flexibilities and program outcomes, we obtained the following 2009–2015 data on MTW and non-MTW agencies: agency and tenant characteristics from the Public and Indian Housing Information Center (PIC) system, public housing occupancy rates from the Picture of Subsidized Households database, voucher unit utilization rates from the Voucher Management System (VMS), and expense data from the Financial Data Schedule (FDS). These were the most reliable and recent data available at the time of our analysis. We combined the HUD data with data from the American Community Survey (1-year estimates) conducted by the Census Bureau. To assess the reliability of these data, we reviewed relevant documentation on the information systems, conducted electronic testing, and interviewed officials knowledgeable about the data. We determined that the data were sufficiently reliable for the purpose of identifying a comparison group and comparing the outcomes of certain measures for MTW and comparable non-MTW agencies. We used these data and multivariate statistical methods to compare MTW and non-MTW agencies to estimate any association between MTW flexibilities and public housing occupancy rates, voucher unit utilization rates, and various public housing and voucher expenses. We used statistical matching and modeling methods to identify a comparison group of non-MTW agencies that closely resembled MTW agencies on characteristics including number of households served, geographic location, and housing market characteristics. For more detailed information on our analysis, see appendix II. To determine the factors that could partially explain the results of our analysis, we reviewed Attachment C of the standard agreement to identify the funding flexibilities the MTW demonstration affords participating agencies. We also reviewed MTW agencies’ 2011–2016 annual plans to identify the MTW activities that were proposed under those funding flexibilities and interviewed officials from the seven selected agencies to learn how they used the funding flexibilities. We started with the 2011 annual plans because that was the first year in which all MTW agencies were required to include specific information when proposing rent-reform policies. We ended with 2016 annual plans because it was the most recent year for which annual plans were available for all MTW agencies at the time of our analysis. To illustrate how MTW agencies used their funding flexibility for public housing, we used FDS data to determine the amount of MTW funds that were transferred from the Housing Choice Voucher (voucher) program to the public housing program. To perform this analysis, we compared the MTW agencies’ 2015 public housing funding—the sum of FDS line items 70600 (HUD public housing agency operating grants) and 70610 (capital grants)—to the aggregate amount MTW agencies transferred into individual public housing project accounts. We selected 2015 because it was the most recent FDS data available at the time of our analysis. We also reviewed 2009–2016 data from HUD on the number of households MTW agencies served through their local, nontraditional activities. We determined that HUD’s process for compiling this information was sufficiently reliable for our purposes of reporting on local nontraditional activities by tracing 2015 data in the spreadsheet to data in the agencies’ 2015 annual reports (the most recent reports available) and interviewing HUD staff. Finally, we analyzed program data that HUD prepared using information derived from the Central Accounting and Program System and VMS on unspent voucher funds as of December 31, 2016, for MTW agencies and the comparison group of non-MTW agencies. To determine the extent to which HUD monitors the effect on tenants of rent-reform, work-requirement, and time-limit policies, we reviewed HUD documents such as Attachment B of the standard agreement and HUD’s Table of Applicable Standard Metrics by Activity to determine how HUD defines these types of activities and the guidance HUD provides on monitoring and reporting their effects on tenants. As previously discussed, we compared HUD’s monitoring policies and procedures with relevant internal control standards. We reviewed MTW agencies’ 2015 annual reports to determine the extent to which agencies adopted rent- reform, work-requirement, and time-limit policies. We selected 2015 because it was the most recent year for which annual reports were available for all MTW agencies at the time of our analysis. We also reviewed agencies’ 2011–2016 annual plans and collected information from all MTW agencies on tools they use to monitor the effects of rent- reform policies on tenants. We reviewed information from all 39 MTW agencies on their hardship policies and data and their annual reevaluations of the impact of rent-reform activities. We also collected information from all MTW agencies on how they monitor the effect of work-requirement and time-limit policies on tenants. We interviewed officials from the seven selected agencies about their monitoring of rent- reform, work-requirement, and time-limit policies’ effects on tenants and associated hardship policies and to obtain their views about HUD guidance. We also conducted group interviews with tenants from five agencies to get their perspective on the effects of rent-reform, work-requirement, and time-limit policies the agencies had implemented and associated hardship policies. To select the tenants to invite to these group interviews, we focused on the populations (for example, those able to work) subject to these policies. To the extent the MTW agency had a resident advisory board or comparable resident association, we worked with the boards or associations to contact tenants. When appropriate, we asked the MTW agencies to post notices on their websites and throughout their properties and send mailings to tenants of interest to notify them about the meetings. Finally, we interviewed representatives from tenant advocacy organizations. The organizations represented tenants served by four of the agencies we visited as well as tenants served by two additional MTW agencies that were not part of the group of seven selected agencies but that also had implemented major rent-reform changes, work-requirement, or time-limit policies. We obtained information on the effect of these policies on tenants and the extent to which tenants were aware of the hardship policies associated with these policies. To select these groups, we generally relied on recommendations from a representative of the National Housing Law Project. For those areas for which a recommendation was not provided, we identified the local legal aid association through an Internet search. We conducted this performance audit from February 2016 to January 2018 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. We analyzed associations between the Moving to Work (MTW) demonstration’s flexibilities and two types of outcomes: housing availability, measured by public housing occupancy and voucher unit utilization rates, and program expenses, measured by public housing operating expenses and voucher administrative, subsidy, tenant services expenses, and voucher reserves per household. These outcomes are broadly consistent with the goals of the demonstration’s authorizing statute. MTW was designed to provide flexibility to participating public housing agencies to design and test innovative strategies, while meeting certain statutory objectives and demonstration requirements, including reducing costs and achieving greater cost-effectiveness and assisting substantially the same number of eligible low-income households as would have been served absent the demonstration. In this appendix, we summarize the statistical methods we used to analyze a dataset we assembled from administrative databases maintained by the Department of Housing and Urban Development (HUD) and the American Community Survey (ACS), conducted by the Census Bureau, to compare MTW and non-MTW agencies on these outcomes. Our analysis did not seek to conduct a definitive evaluation of the MTW demonstration’s causal impacts. MTW agencies carry out varied and unique activities. The agencies also vary widely in size, location, housing market, and area and tenant demographics—both compared to non-MTW agencies and among themselves. A persuasive impact evaluation would need to assess the unique circumstances of each activity and outcome at each agency. In contrast, our analysis sought to improve on simple comparisons of outcomes between MTW and non-MTW agencies, by constructing a comparison group of non-MTW agencies that were similar to MTW agencies on variables broadly relevant to housing programs. Although this multivariate analysis reduced the risk that factors other than MTW participation may have biased the comparison, we did not seek to hold constant all factors uniquely relevant to each MTW agency and activity. As a result, our analysis cannot provide definitive estimates of causal impacts. Developing and applying statistical “treatments” to MTW agencies is complex, due to demonstration rules that allow agencies to conduct various activities tailored to their unique needs. We considered the option of forming several groups of MTW agencies, defined by similar activities. For example, we might have identified all agencies reforming HUD’s rent calculation formula, and included those agencies in one level of a multilevel treatment variable. We ultimately rejected this approach due to limited sample sizes and the difficulty of developing homogeneous groups of activities. A multilevel approach would have limited the number of agencies in each level of the treatment. Small sample sizes would have limited our statistical power to identify differences between treatment groups, if they existed. In addition, the wide variety of MTW activities would have made it difficult to produce a sufficient number of homogenous groups, and would have required subjective judgment about what activities were sufficiently similar. Instead, we used a binary treatment measure identifying agencies that participated in MTW or operated under traditional public housing rules in a given year. The timing of MTW implementation limited our ability to account for changes in participation and outcomes over time. Agencies joined the MTW demonstration at various times between 1996 and 2012, and many joined before sufficient data became available. Only nine agencies entered the demonstration after 2009, when HUD’s Public and Indian Housing Information Center (PIC) system began to provide sufficiently complete and reliable data on the characteristics of housing agencies we needed to measure. All agencies that exited the demonstration did so before 2009. Comparisons within agencies over time can implicitly control for other factors that may not substantially change before and after implementation by using data collected before and after agencies joined the MTW demonstration. We might have been able to implicitly control for many factors that did not substantially change over short periods, such as land prices, or that changed in identical ways for MTW and non-MTW agencies, such as national economic cycles. However, the implementation of the MTW demonstration and available data limited our analysis to repeated cross-sectional comparisons of MTW and non-MTW agencies from 2009 through 2015. Measuring participation in the MTW demonstration at any one time was somewhat imprecise. The MTW demonstration was not implemented at uniform times across agencies, due to variation in the ratification dates of MTW agreements between HUD and the agency and variation in when each MTW agency began to implement activities under the demonstration. For our primary analysis, we classified an agency as participating in the MTW demonstration if it had ratified an MTW agreement with HUD at least 1 year before the year measured. In sensitivity analyses, described below, we assessed how classifying MTW participants according to different time lags affected our results. Table 6 lists the number of MTW and non-MTW agencies in our dataset, based on how MTW participation was defined in the analysis for housing agencies in the PIC database from 2009 through 2015. We compared MTW and non-MTW agencies on several outcomes that are broad measures of housing availability and expenses. The outcomes were available in HUD data systems and were reliable for our purposes. However, they do not exhaust the potential outcomes that may be relevant under the MTW authorizing statute or the design of specific agency activities. For example, potential outcomes could measure the number of households that achieve self-sufficiency (as defined by a MTW agency) or move to a low-poverty neighborhood. Our specific outcome measures were the following: Public housing occupancy rate. Occupied units as a percentage of units available. Voucher unit utilization rate. Monthly rate of unit months leased divided by unit months available for the public housing agency. Public housing operating expenses per household. Total yearly operating expenses, divided by number of public housing households. Public housing central office cost center expenses per household. Total yearly central office cost center operating expenses, divided by number of public housing households. Voucher administrative expenses per household. Total yearly administrative expenses, divided by the number of voucher households. Voucher subsidy expenses per household. Total yearly expenses for housing assistance payments, divided by the number of voucher households. Voucher tenant services expenses per household. Total yearly expenses for tenant services, divided by the number of voucher households. Reserves per household (2016 only). Unspent voucher housing assistance funds as of December 31, 2016, divided by the number of voucher households. Following the Rubin Causal Model, our primary parameter of interest was the average (or median) treatment effect on the treated: where Yij(T) denotes the outcome for agency i at time j in (potentially counterfactual) treatment condition T. That is, we estimated the expected difference in outcomes that would exist due to MTW participation, among those agencies that actually participated in the demonstration. Estimating the average treatment on the treated is conservative and appropriate, given the varied and unique nature of MTW activities. Generalizing the effect of MTW participation from the treated agencies to the rest of the public housing agency population makes the implausible assumption that the untreated agencies would have implemented the same activities, in the same ways, as the treated agencies. Due to the discretion inherent to the MTW demonstration, the experiences of the treated agencies may not generalize to the whole population, as would be required for estimating the average treatment effect. We specify a parameter of interest (that is, a value to be estimated) for methodological completeness and to specify the population of inference (the target population of agencies). However, we do not interpret our results as robust causal impact estimates, due to the inability to measure the unique circumstances relevant for each MTW agency, demonstration activity, and outcome. Our analysis measured and held constant conditions that could have otherwise explained differences in outcomes between MTW and non- MTW agencies. For each MTW and non-MTW agency, we measured the following agency-level covariates (with sources in parentheses): Number of households (PIC) Percent of households with a member over the age of 65 (PIC) Percent of households with a member under the age of 18 (PIC) Percent of households with a disabled member (PIC) Whether an agency issues vouchers (VMS) County median household income (ACS) County median rent (ACS) County rental vacancy rate (ACS) County population density, measured as county population/land area (2010 Census) HUD region (HUD website) Latitude (Picture of Subsidized Households) Longitude (Picture of Subsidized Households) We assessed the reliability of the ACS estimates by calculating the ratio of each estimate’s 95 percent margin of error to the estimate. For example, this ratio would equal 5 for an estimated rental vacancy rate of 10 percentage points, with a margin of error equal to plus or minus 2 percentage points. Across all variables we used from ACS, we found that this ratio did not exceed 2.0 for 99 percent of agency-county observations. This level of reliability was acceptable for our purposes. When PIC showed that agencies spanned multiple counties, we aggregated the data to the agency level by either summing count variables across counties or calculating averages of ACS descriptive statistics, such as county mean incomes. We calculated unweighted averages because the Census Bureau does not release ACS microdata with the exact geographic locations needed to re-estimate the statistics of interest within public housing agency boundaries. Weighting by the total area population or number of households served by each public housing agency would have had unknown effects on the bias of the published ACS estimates, due to their complex weighting methods. Our aggregation methods should minimally influence our measurements, due to limited variation across counties within agencies. To quantify this variation, we calculated the coefficient of variation (CV) across counties served by each agency in our analysis, and these CVs of the ACS statistics did not exceed 0.99 for 50 percent of the agencies and 1.73 for 95 percent of the agencies. We used statistical matching methods to construct the comparison group of non-MTW agencies. The general iterative matching process involves 1. identifying some distance measure that quantifies how “close” units are to each other on the covariates of interest; 2. implementing a matching method that uses this distance measure to identify comparison units; and 3. assessing the quality of the matched samples and iterating between the first two steps, until the treatment and comparison groups become sufficiently close on the distance measure. We developed our specific matching approach using recent reviews of the statistical literature. Two established matching methods rely on propensity scores and Mahalanobis distance (MD). In the context of this analysis, propensity scores estimate the probability that an agency is an MTW or non-MTW agency, such as when Pr(MTW | X) = logit-1(Xβ), where X is a matrix of covariates and β is a vector of coefficients. Propensity scores are calculated using the estimated coefficients and X to obtain a predicted probability that an agency participates in the MTW demonstration. MD is a multivariate sample statistic measuring the distance between agency i and j, similar to the number of standard deviations away from the sample mean vector of the covariates: where X is the ith row vector of X and S is the sample covariance matrix. Propensity scores and MD measures can have several limitations in practice. Matching on known propensity scores is used to balance the covariate distributions between the treatment and comparison groups and matching using MD tends to improve balance across all measured covariates. However, both approaches are optimal under assumptions of normally distributed data, and may worsen covariate balance if this assumption does not hold. Genetic matching methods seek to solve the problem of achieving sample balance in practice, using computer algorithms to search over the space of possible distance measures. Genetic matching generalizes MD by weighting covariates according to how they achieve balance in any particular sample, rather than by constants equal to the inverse of their sample covariance matrix, as in MD: where W is the covariate weighting matrix. If desired, genetic matching can incorporate propensity scores by including them as a covariate, with the algorithm assigning as much weight to them as necessary to optimize balance. The genetic matching algorithm, as implemented by the R software package “Matching,” has the following steps: 1. Initialize covariate weights, W, at starting values. 2. Calculate the distance matrix between MTW and non-MTW agencies. 3. Specify the number of non-MTW agencies to be matched comparison agencies for each MTW agency. 4. Assess the balance between the sample distributions of the treatment and control groups, using p-values from matched t-tests of equal means for each covariate or Kolmogorov–Smirnov tests of equal distributions. 5. Apply a loss function to the vector of p-values to quantify overall sample balance. 6. If the loss function is not minimized, regenerate W using a genetic algorithm. 7. Repeat steps 2–6 until the loss function is optimized and covariate balance is maximized. In sum, the genetic matching algorithm searches for the best k matches, incorporating covariates and distance metrics as desired and minimizing the distance in a candidate matched set by weighting and reweighting the covariates and metrics, according to how they influence balance. In our primary analysis, we ultimately used one-to-one matching (k = 1), with one comparison agency selected for each MTW agency. Large imbalances in the number of households served by the MTW and non- MTW agencies substantially reduced the pool of similar comparison agencies, such that setting k > 1 substantially worsened the balance for some variables. In addition to the automated matching criteria above, we compared the sample distributions of the covariates before and after matching using descriptive statistics and nonparametric density estimates. We required exact matches on the year of measurement to ensure that observations were compared at roughly the same times. We also required exact matches on whether an agency issued vouchers and HUD region. Due to data limitations, we compared 2016 reserve spending between MTW and non-MTW agencies for the 2015 matched set. Figure 12 compares MTW agencies and non-MTW agencies on the covariates we identified, before constructing a matched sample of comparable non-MTW agencies. As the figure shows, there are some covariates for which there are significant differences between the group of MTW agencies and non-MTW agencies. After implementing the matching method described above, we identified a primary group of comparison agencies that were similar to the MTW agencies on most of the covariates, but differed on a few, as shown in table 7. Examples of matched agencies in our primary analysis include: Oakland Housing Authority (MTW) and Housing Authority of the County of Sacramento (non-MTW); San Antonio Housing Authority (MTW) and Housing Authority of New Orleans (non-MTW); and Housing Authority of the City of Pittsburgh (MTW) and Allegheny County Housing Authority (non-MTW). Imbalances between MTW and comparison agencies for the main analyses remained after our primary matching analysis for county median income, county median rental cost, number of households, percent of households with a disabled member, and county rental vacancy rate, as shown in table 7. Figure 13 shows the covariate density estimates for MTW and non-MTW agencies, after matching. As the figure shows, there are fewer differences in the group of MTW agencies and the matched non-MTW agencies after matching. MTW agencies had higher county median incomes and rent, lower percentages of disabled household members, and lower rental vacancy rates, as compared to the primary matched non-MTW agencies. These imbalances decreased when we allowed for matches across HUD region and required matches within calipers (1 standard deviation), as shown in table 8. However, allowing HUD region to vary potentially allowed other unmeasured factors within a HUD region to vary between the MTW and non-MTW groups. Applying caliper constraints failed to match a comparison agency for 91 of the 232 yearly observations for MTW agencies during 2009–2015, which changes the population for inference. We used these matched samples with improved balance for sensitivity checks, in our discussion of the results below. After constructing the primary matched analysis sample, we estimated outcome descriptive statistics for MTW and non-MTW agencies. We estimated differences in mean and median outcomes using paired t-tests and nonparametric Wilcoxon signed-rank tests, respectively, that account for correlations over time within and between matched groups of MTW and non-MTW agencies. We estimated differences in medians between groups using nonparametric Wilcoxon signed-rank tests to address potential outliers. For example, the tenant services cost distributions for MTW agencies (median = $37; 25th quantile = $2.80; 75th quantile = $110) and non-MTW agencies (median = $0; 25th quantile = $0; 75th quantile = $20) were highly skewed. The nonparametric test was not influenced by these skewed distributions and outliers. To complement this matched comparison, we used Generalized Linear Models to model outcomes in 2009–2015 using the matched sample of MTW and non-MTW agencies. The models had the form: i = 1, …, n indexes agencies j = 2009, …, 2015 indexes years MTWij indicates whether agency i participated in the MTW demonstration µij is the mean outcome, conditional on the covariates g is the Gaussian link function Year is a vector of indicators for each year from 2010 through 2015 (excluding 2009), which accounts for common period effects across agencies, γ Xij is a vector of linear continuous (e.g., number of households) and categorical (e.g., HUD region) control variables that may confound the association between agency type and the outcome of interest (discussed above for the matched sample) β is the parameter of interest, estimating the association between MTW Repeated observations from 2009 through 2015 for MTW agencies and their corresponding matched non-MTW agencies can introduce autocorrelation within these clusters of observations, and the differences across matched clusters can introduce heteroscedasticity (that is, the variance in one cluster of agencies may be not be consistent with the variance in another cluster). A conventional linear model does not account for these interdependencies and inconsistent variances in the data, leading to potential bias in the variance estimation for the parameters of interest (such as variances for β and γ) and any subsequent statistical inference on the association (and p-values) between the outcome and covariates. To account for the potential bias arising from heteroscedasticity and autocorrelation, the variance-covariance matrix used to generate the variances for the parameters incorporated weights that (1) decreased the influence of extreme observations, clusters, or both; (2) used an autoregressive approximation in which the correlation was strongest for observations closest in time and decays as time lengthens; and (3) preprocesses (“prewhitens”) the variance-covariance matrix using an autoregressive function to reduce the temporal dependence in the data. These processes lead to statistical inference on associations of interest that account for the interdependencies within agency clusters and the differences across clusters. In the sensitivity analyses described below, we will fit this model on the unmatched population of agencies. In the matched sample, MTW agencies had lower median public housing occupancy rates and voucher unit utilization rates compared to non-MTW agencies, as shown in table 9. Compared to non-MTW agencies, MTW agencies had higher median public housing expenses per household (operating and central office cost center operating expenses) and higher median voucher administrative expenses per household, subsidy expenses per household, tenant services expenses per household, and reserves per household. These differences were significant at the 0.05 level for all variables using the nonparametric Wilcoxon signed-rank test. However, using the parametric t-tests and related t-tests from the regression models, there was not a significant difference in central office cost center operating expenses. This could arise from the presence of outliers skewing the distribution, leading to different results compared to the Wilcoxon test that does not make any distributional assumptions. Regardless of the particular method used, small sample sizes in each group, as well as repeated observations over time, may limit our statistical power to identify differences, if they existed. Sample sizes resulting from missing data also affect the degree to which comparable non-MTW agencies can be found, given the limited overlap in the covariate distributions between groups. We assessed the results above for sensitivity to various methodological assumptions. For the matching analysis, we assessed the impact of 1. measuring MTW status as of the agreement year, rather than as of 1 year following the agreement (i.e., 1 year lag); 2. matching within 1 standard deviation calipers for each covariate; 3. allowing matches between HUD regions; 4. including county unemployment and poverty rates as covariates; 5. including estimated propensity scores, as a logistic function of the control variables described for the primary analysis, as a matching covariate; 6. increasing the number of comparison agencies for each MTW agency to k = {2, 3, 4} using the control variables described for the primary analysis; and 7. excluding clusters where the MTW and/or non-MTW agencies had an outlying value for an outcome of interest. For the regression model, we compared the results obtained from fitting the model to the matched and unmatched data. The sensitivity tests above showed no substantively meaningful differences in the results as compared to the primary analysis, with several exceptions. Adding the caliper constraint and dropping the HUD region constraint improved covariate balance. Dropping the HUD region constraint led to MTW agencies having a smaller difference in voucher subsidy expenses, compared to non-MTW agencies. In our primary analysis, MTW agencies had higher subsidy expenses. However, allowing matches between HUD regions may introduce unmeasured geographic characteristics into the comparison group of non-MTW agencies, which may limit the comparability of subsidy expenses and bias the estimated difference in outcomes. In addition to the contact named above, Paige Smith (Assistant Director), Josephine Perez (Analyst in Charge), Enyinnaya David Aja, Bethany Benitez, Farrah Graham, Anar N. Jessani, Morgan Jones, Courtney LaFountain, Won Lee, Marc Molino, Anna Maria Ortiz, Barbara Roesmann, Shannon Smith, and Jeff Tessin made key contributions to this report.

Summary: The MTW demonstration gives 39 participating public housing agencies the flexibility to use funding for HUD-approved purposes other than housing assistance, such as developing affordable housing; change HUD's tenant rent calculation; and impose work requirements and time limits on tenants. In 2015, Congress authorized the expansion of MTW by adding 100 new agencies. GAO was asked to evaluate the MTW demonstration. GAO examined HUD oversight of MTW agencies, including its monitoring of demonstration effects on tenants. For this report, GAO reviewed HUD and MTW agency policies and documentation; interviewed officials at HUD and seven MTW agencies (selected based on type of policy changes, size, and geographic diversity); and interviewed tenants served by selected agencies. GAO also conducted a statistical analysis comparing data for MTW and non-MTW agencies on public housing occupancy rates, voucher utilization rates, and program expenses. The Department of Housing and Urban Development‘s (HUD) oversight of the Moving to Work (MTW) demonstration has been limited. Improving oversight—particularly for information collection and analysis—would help HUD assess what MTW agencies have done, including funding use. HUD took steps to improve oversight and reporting, but GAO found limitations in the following areas: Workforce planning. While HUD has taken steps to address staffing to oversee the current 39 MTW agencies, HUD has not finalized its workforce planning for 100 agencies to be added to the demonstration. According to a 2015 HUD analysis, a large number of additional staff would be needed for the expansion. HUD officials said field office staff might assume greater oversight responsibilities to fill this gap, but a joint (headquarters-field) oversight structure is not final and HUD's workforce analysis has not been updated to reflect this proposed oversight structure. Data collection. Due to limited data, HUD cannot fully determine the extent to which demonstration flexibilities affected the performance of MTW agencies, especially in relation to outcomes that affect the number of tenants served—occupancy and voucher utilization rates and program expenses. GAO found that MTW agencies had lower yearly median rates for public housing occupancy and Housing Choice Voucher (voucher) unit utilization and higher yearly median program expenses than comparable non-MTW agencies. The differences may be partly the result of demonstration funding flexibilities, such as the ability to use public housing and voucher funding for purposes such as gap financing for affordable housing (a nontraditional activity). But limitations in HUD data (such as not differentiating expenses for nontraditional activities) make it difficult to fully explain differences in outcomes GAO analyzed. Oversight of reserves. HUD has not implemented a process to monitor MTW reserves or agencies' plans for such reserves, which led to agencies accruing relatively large amounts of unused funds that could be used for vouchers. According to HUD data as of June 30, 2017, the 39 MTW agencies had more voucher reserves than the 2,166 non-MTW agencies that administer the voucher program combined ($808 million compared to $737 million). Without a monitoring process, HUD cannot provide reasonable assurance that MTW agencies have sound plans for expending reserves. Monitoring the effect of rent reform, work requirements, and time limits on tenants. HUD is limited in its ability to evaluate the effect of MTW policies on tenants. HUD does not have a framework—including clear guidance on reporting requirements and analysis plans—for monitoring the effect of rent-reform, work-requirement, and time-limit policies. HUD guidance instructs agencies to analyze the impact of their rent reform activities, describe how they will reevaluate them, and develop a tenant hardship policy for such policies (but not for time limits or work requirements). But the guidance does not describe what must be included in the analyses or policies, leading to wide variation in how agencies develop them. Also, HUD does not assess the results of agencies' analyses. GAO makes 11 recommendations to HUD, which include completing workforce planning, developing processes to track use of funds and monitor agencies' reserves, and developing a framework—including clear guidance on reporting requirements and analysis plans—to monitor effects on tenants. HUD generally agreed with eight of the recommendations and disagreed with three, citing the need for flexibility. GAO maintains the recommendations, as discussed further in the report.