Mortgage servicers are the entities that manage payment collections and other activities associated with mortgage loans. Servicing duties can involve sending borrowers monthly account statements, answering borrowers’ inquiries, collecting monthly mortgage payments, and maintaining escrow accounts for property taxes and insurance. In the event that a borrower becomes delinquent on loan payments, servicers also initiate and conduct foreclosures. Errors, misrepresentations, and deficiencies in foreclosure processing can result in a number of harms to borrowers ranging from inappropriate fees to untimely or wrongful foreclosure. A number of federal regulators share responsibility for regulating the banking industry in relation to the origination and servicing of mortgage loans. OCC has authority to oversee nationally chartered banks and federal savings associations. The Federal Reserve oversees insured state-chartered banks that are members of the Federal Reserve System, bank and thrift holding companies, and entities that may be owned by federally regulated depository institution holding companies but are not federally insured depository institutions. In September 2010, allegations surfaced that several servicers’ documents accompanying judicial foreclosures may have been inappropriately signed or notarized. In response to this and other servicing issues, federal banking regulators directed servicers to complete self-assessments of their foreclosure processes. In addition, banking regulators conducted a coordinated on-site review of 14 of the largest mortgage servicers to evaluate the adequacy of controls over servicers’ foreclosure processes and to assess servicers’ policies and procedures for compliance with applicable federal and state laws. On the basis of their findings from the coordinated review, OCC, OTS, and the Federal Reserve issued in April 2011 formal consent orders against the 14 servicers under their supervision. To comply with the consent orders, each of the 14 servicers is required to, among other things: enhance its vendor management, training programs and processes, and compliance with all applicable federal and state laws, rules, regulations, court orders, and servicing guidelines. In addition, the consent orders required each servicer to retain an independent firm to review certain foreclosure actions on primary residences from January 1, 2009, to December 31, 2010. Third-party consultants were permitted to retain outside counsel to provide necessary legal expertise in completing the foreclosure review. Through the foreclosure review, consultants were to identify borrowers who suffered financial injury as a result of errors, misrepresentations, or other deficiencies in foreclosure actions, and recommend remediation for harms suffered by borrowers, as appropriate. In general, the consent orders identified seven areas for consultants to review: (1) whether the servicer had proper documentation of ownership of the loan; (2) whether the foreclosure was in accordance with applicable state and federal laws; (3) whether a foreclosure sale occurred while a loan modification was under consideration; (4) whether nonjudicial foreclosures followed the terms of the loan and state law requirements; (5) whether fees charged to the borrower were permissible, reasonable, and customary; (6) whether loss-mitigation activities were handled in accordance with program requirements and policies; and (7) whether any errors, misrepresentations, or other deficiencies resulted in financial injury to the borrower. Servicers proposed third-party consultants to conduct the foreclosure reviews. After regulators reviewed the independence of proposed third-party consultants, servicers and third-party consultants submitted engagement letters outlining their foreclosure review processes to the regulators for their review and approval. OCC and the Federal Reserve posted approved engagement letters between the servicers and third-party consultants on their respective websites. Regulators also posted on their websites a financial remediation framework that provided examples of errors covered by the consent orders and the corresponding compensation or other assistance that consultants could recommend based on their findings. The foreclosure review had two components: a process for eligible borrowers to request a review of their particular circumstances (referred to as the borrower outreach process) and a review of categories of files (referred to as the look-back review). The borrower outreach process was intended to complement the look-back review and help identify borrowers who may have suffered financial injury. The regulators required the servicers to inform borrowers who believed they might have been financially harmed due to inappropriate foreclosure that they could submit a request for review of their particular circumstances. The servicers conducted this outreach through advertising and direct mail. After several extensions, the final deadline for submission of these requests was December 31, 2012. For the look-back review, the consent orders allowed third-party consultants to use statistical sampling techniques to select samples of files for review from various categories of loans, pursuant to regulators’ guidance and approval as well as 100 percent review of certain loan categories. Regulators established minimum sampling requirements for third-party consultants to conduct statistically valid sampling of the target population and expectations for additional sampling methods, if warranted, to identify as many harmed borrowers as possible for remediation. Bank examiners often use sampling methods to review the files of a financial institution’s operations. The use of sampling by examiners has been a regularly accepted practice and is commonly used as a means to review a bank’s files when a full review of all files is not practicable. OCC states in its handbook on sampling methodologies, which is geared toward sampling loan portfolios, that it is impractical or impossible for bank examiners to review all items or files when examining an area of a bank’s operations, especially if the volume of information is large. Examiners use sampling to identify a random subset of files to learn about the multitude of items from which those files are drawn. Upon drawing appropriate statistical inferences from this subset, they can state with a certain level of confidence that the inferences apply to the population as a whole. The benefits of using statistical sampling include the ability to generalize results to the sampled populations and to quantify uncertainty in estimates attributable to sampling. In our prior report on the borrower outreach component of the foreclosure review, we found that regulators and servicers had gradually improved the communication materials for borrowers but that regulators could make further enhancements to the outreach efforts. First, we found that regulators did not consider best practices, such as using tests or focus groups, to assess the readability of the outreach materials and did not solicit input from consumer groups when reviewing initial communication materials. As a result, we reported that the materials at that time might have been too complex to be widely understood. Second, we found that, although the communication materials included information about the purpose, scope, and process for the foreclosure review and noted that borrowers may be eligible for compensation, the materials did not provide enough specific information about remediation, which best practices suggest could have encouraged more borrower responses. Third, we found that the outreach planning and evaluation targeted all eligible borrowers with limited analysis conducted to tailor the outreach to specific subgroups within the population. Therefore, we noted that some underrepresented borrowers may not have been apparent to regulators without further analysis of the characteristics of respondents compared to nonrespondents. To help ensure that all borrowers had a fair opportunity for review, we recommended that OCC and the Federal Reserve enhance the language on the foreclosure review website, include specific remediation information in the outreach, and require servicers to analyze trends in borrowers who have not responded and, if warranted, take additional steps to reach underrepresented groups. In response to the recommendations that we made in our previous report, OCC and the Federal Reserve took steps to enhance the independent foreclosure review website and communication materials and conducted more targeted outreach. We determined that regulators implemented our first recommendation by including on the foreclosure review website a help sheet for the request-for-review form that provides tips in plain language, an explanation of key terms, and additional instructions to help borrowers fill out the form. In response to our second recommendation, OCC and the Federal Reserve publicly released a financial remediation framework and included ranges of potential payment amounts or categories in their communication materials and other outreach. For example, the regulators released 60-second radio and television advertisements stating that if consultants find errors, homeowners may be eligible for compensation or other remedies, such as refunding fees, stopping a foreclosure action, or making payments that could range from $500 to $125,000. In addition, at the regulators’ instruction, servicers included similar language on ranges of potential payment amounts or categories in other outreach materials. In response to our third recommendation, OCC and the Federal Reserve tailored their recent outreach actions to target communities based on audience characteristics, response data, and consumer research. Regulators included a wide range of minority media and a broad mix of media formats, including print advertisements, radio, television, and Internet. To tailor this outreach, a market analysis was completed to identify areas and ethnic groups with the greatest opportunity for increased awareness. Outreach materials also were made available in eight different languages. Regulators solicited numerous community groups for their input and assistance on the outreach and identified effective messengers by using leaders of community groups that represent minorities to deliver radio and television public service announcements. In January 2013, OCC and the Federal Reserve issued joint press releases stating that they had reached agreements with 11 of the 14 mortgage servicing companies subject to the April 2011 consent orders to discontinue the foreclosure review conducted by third-party consultants and to provide almost $8.8 billion in cash payments and foreclosure- prevention assistance to borrowers. With this change in direction from the foreclosure review to an agreed-upon payment process, regulators and servicers moved away from identifying the types and extent of harm borrowers may have experienced and focused instead on assigning borrowers into categories based on objective criteria and issuing payments in what they expect will be a shorter amount of time than would have occurred under the foreclosure review. To explain their rationale for pursuing the agreements, OCC and Federal Reserve staff said they considered a variety of factors, including delays in payments to harmed borrowers, the total remediation payments expected to be made to borrowers, and costs of the reviews. Under the agreements, servicers will provide approximately $3.4 billion in direct payments to eligible borrowers. Using a framework provided by regulators and characteristics of borrowers’ loans, servicers will categorize borrowers, and regulators will develop a distribution plan and direct a payment administrator to distribute cash payments. As a result, all borrowers who were eligible for foreclosure reviews under the consent orders are expected to receive payments ranging from hundreds of dollars up to $125,000, depending on the borrower’s category. Under the agreements, servicers will also provide approximately $5.4 billion in foreclosure-prevention assistance to borrowers, such as loan modifications. To the extent practicable, servicers are to prioritize such assistance for borrowers eligible for the foreclosure review. Eligible borrowers are expected to receive notice about the payments by the end of March 2013, whether or not they filed a request-for-review form, and borrowers will not need to take further action to be eligible for compensation. Nearly 4 million borrowers, or about 90 percent of the eligible borrower population, are covered by the servicers that signed the agreements. In late February 2013, regulators released amendments to the April 2011 consent orders that incorporated the provisions of the agreements. The amended orders are publicly available on the regulators’ respective websites. Consultants for the servicers that did not reach agreements with the regulators—Ally Financial (GMAC Mortgage), Everbank, and OneWest—continue their foreclosure review activities. More than 450,000 borrowers fall under the three servicers that do not have amended consent orders. In April 2011, regulators entered into consent orders with 14 mortgage servicers. The consent orders require the servicers to conduct foreclosure reviews by engaging third-party consultants to review the servicers’ loan files to identify and remediate the financial injuries suffered by borrowers through errors, misrepresentations, or other deficiencies in their foreclosure processes in 2009 and 2010. In January 2013, the regulators announced agreements in principle with 11 of the 14 servicers subject to the April 2011 consent orders. The parties agreed to replace the foreclosure review with a compensation framework that does not rely on determinations of whether borrowers suffered financial harm, instead requiring participating servicers to provide cash payments and foreclosure-prevention assistance to borrowers. In February 2013, the regulators publicly released amended consent orders that formally replaced the requirements related to the foreclosure review for the servicers participating in the agreements. According to regulators, the goals of the foreclosure review were for consultants to identify as many harmed borrowers as possible, to treat similarly situated borrowers across all 14 servicers similarly, and to help restore public confidence in the mortgage market. However, regulators faced various challenges in accomplishing these goals. According to regulator staff and third-party consultants, coordinating the foreclosure review process was challenging because of the large number of actors directly involved in the review process and the large number of borrowers eligible for the review. Specifically, the foreclosure review process included 14 servicers; 14 third-party consultant teams from 7 different consulting and accounting firms, some with subcontractors; and more than 10 third-party law firms. Further, the reviews were overseen by local examination teams for OCC or the Federal Reserve as well as headquarters staff for both regulators. In addition, there were as many as 4.3 million borrowers whose files could have been reviewed. One examination team informed us that the size of the review population presented challenges beyond those that accompany a typical consent order. Another challenge examiners and consultants noted was that, due to the deadline extensions for borrowers to submit their requests-for- review, the volume of the file review work kept changing and the total number of files requiring review was unknown. The original deadline for requests-for-review was April 2012. Regulators reported that more than 160,000 borrowers had submitted requests-for-review as of April 30, 2012. The deadline was extended three times to December 31, 2012. Regulators reported that more than 510,000 requests were submitted and received as postmarked on or before the December 31, 2012 deadline. According to OCC, roughly 300,000 additional files were selected for review through the consultants’ look-back review process. OCC staff told us that more files likely would be selected if errors were found and consultants needed to conduct additional sampling to identify as many harmed borrowers as possible. Third-party consultants and law firms told us that the size of the loan files and the scope of the file review made the process complicated and time- consuming. Consultants from whom we obtained information told us that a typical loan file is large with many types of documents. For example, documents may include servicer notes on communication with the borrower, documents collected from the state foreclosure attorneys responsible for the foreclosure activities, records of fees charged and payments made, and, in many cases, documents assessing a borrower’s eligibility for loan modification and loss mitigation activities. Consultants told us that some files may contain as many as 50 documents, potentially comprising more than 2,000 pages. Consultants also stated that the reviews were challenging because they covered such a wide variety of complex issues, including different state foreclosure laws, federal laws and regulations, and guidelines for federal and servicers’ proprietary loan modification programs. To assess each of these areas, consultants developed a series of test questions—generally yes or no questions—to identify potential errors. The number of test questions used by third-party consultants to conduct the file reviews varied. For example, one consultant told us that they had approximately 2,600 test questions with more than 4,000 discrete steps, while another consultant told us they had 16,000 test questions. Further, third-party consultants from whom we obtained information stated that their reviewers spent as many as 50 hours to complete a full file review, although review times varied depending on the issue and type of review. OCC and the Federal Reserve stated that the uniqueness of each servicer’s borrower population and process for recording and storing information on borrowers’ loan files posed challenges for defining the review parameters and developing a uniform review structure for all the consultants. For example, Federal Reserve staff told us that servicers’ systems have different ways to identify borrowers who are current on a loan modification and that the systems have varying capabilities to provide information about the last payment received from a borrower. As a result, consultants had to take different approaches to identify borrowers with the same characteristics. In addition, Federal Reserve staff told us that servicers had different concentrations of loans in geographic areas and that, therefore, consultants might select samples differently based on these concentrations. For example, if a servicer did not have high numbers of foreclosures in a particular state, choosing a large sample of loans in that state would not have been appropriate. As a result, OCC and Federal Reserve staff said that it was not feasible to design one file review process that would apply to all servicers and that it was necessary for third-party consultants to tailor their file review processes, particularly the review questions used to conduct the file review, to the unique characteristics of a given servicer. Faced with various complexities and challenges that the foreclosure review posed, regulators told us that they issued guidance and took a number of oversight steps. First, the sections of the consent orders issued to servicers supervised by both OCC and the Federal Reserve outlining the purpose of the foreclosure review were nearly identical. According to OCC and Federal Reserve staff, the similarity in the consent orders was intended to ensure that the reviews covered the same issues and resulted in similar results for similarly situated borrowers. Consultants we interviewed said that they designed their reviews to address the issues as they were identified in the consent orders. Second, regulators issued guidance to third-party consultants to help frame the file review process and promote consistency in its implementation. Between May 2011 and October 2012, regulators issued 29 joint pieces of guidance to third-party consultants on various topics. For example, OCC and the Federal Reserve jointly issued a financial remediation framework that was designed to be a unifying factor among all the reviews by helping to ensure that similarly harmed borrowers received similar remediation. Regulator staff said that they issued guidance in response to similar questions they received from multiple consultants or examination teams, which oversaw the reviews at the local level. For example, one third-party consultant we interviewed said that the reviews of issues related to the Servicemembers Civil Relief Act (SCRA) would likely provide fairly consistent results for borrowers due, in part, to the clear guidance provided by regulators. In addition to consistent consent orders and guidance, OCC and Federal Reserve staff implemented regular communication mechanisms to help foster consistency in the reviews. Regulators had a robust system of regular meetings involving third-party consultants, servicers, examination team staff overseeing the consultants’ work, and OCC headquarters and Federal Reserve Board staff to discuss challenges with the file review process and help promote consistency among the reviews. OCC and the Federal Reserve met with the parties both separately and as a group and received weekly status reports from the consultants. For example, regulator staff said that weekly calls with third-party consultants provided consultants with an opportunity to raise areas of inconsistency that they had identified. These calls were also used to disseminate new guidance and discuss comments on any pending guidance. According to third-party consultants and examination team staff we interviewed, these meetings were helpful for sharing information among the reviews, developing a similar understanding of the file review process, and discussing challenges consultants encountered in reviewing files that may have affected the consistency of the results for borrowers. In addition to meeting with consultants and servicers, OCC and Federal Reserve staff also held a separate weekly meeting, which included only headquarters staff of each regulator, to discuss new and emerging issues or requests for clarification on guidance that each regulator received from its respective local examination teams. In addition to the weekly calls, OCC headquarters staff and Federal Reserve Board staff visited each of the consultants to observe the file review processes. The April 2011 consent orders expressly allowed third-party consultants to use sampling techniques to identify harmed borrowers. According to regulator staff, the large number of loans in the review population as well as the complexity of the file review process made it difficult for consultants to review all the eligible loan files for errors. As a result, the review relied on sampling, a process of selecting units—in this case, foreclosure files—in a manner that regulators envisioned would allow consultants to identify patterns in errors. In May 2011 regulator-issued guidance on sampling, regulators expressly allowed third-party consultants to use sampling and noted that the consultants’ sampling methodologies should take into consideration public perception as well as the need to provide a high degree of certainty that borrowers who were financially harmed would be identified and obtain remediation. The May 2011 guidance outlined broad parameters for how third-party consultants should approach the sampling methodology to identify harmed borrowers and support a consistent review, such that similarly situated borrowers would have similar results. As part of sampling, regulators anticipated that third-party consultants would segment the review population into loan categories as some loan categories had a potentially higher likelihood of errors. For example, the guidance identified some potential high-risk loan categories, including certain states, foreclosure law firms, and servicing or foreclosure processing activities (e.g., rescinded foreclosures or foreclosure that occurred after loan modification) that could be associated with a higher likelihood of servicing or foreclosure-related errors. Regulator staff stated that differences among the servicer’s loan portfolios and servicing practices made requiring specific loan categories for review inappropriate, with the exception of three categories in which regulators anticipated that consultants would identify a relatively large number of errors in the files. For these three categories—bankruptcies, SCRA loans, and agency- referred foreclosure cases—regulators required 100 percent review of files. The May 2011 guidance also indicated that consultants should be prepared to conduct a second stage of additional analysis of files with a certain number of servicing and foreclosure-related errors that were identified during the initial sampling. Based on our analysis of the sampling plans developed by third-party consultants, we found that the sampling methodologies used by consultants varied among the reviews. Expected population error rate. The sampling plans varied in their expected population error rate from 0 percent—that is, consultants expected to find few or no servicing or foreclosure processing errors in the sampled loan category—to 10 percent, with 8 of the 14 reviews assuming an error rate of zero, three assuming an expected population error rate of 3 percent, and three a rate of 10 percent. Loan category sample size. As a result of different expected population error rates, the plans varied in the size of the samples for analyzing various loan categories identified by the consultant. For example, some consultants selected approximately 100 loans in a sampled loan category for a review, and some consultants selected approximately 370 loans in the sampled loan categories. Loan categories. Based on our analysis, the loan categories used by consultants for their analysis varied from review to review. For instance, although all the consultants analyzed the files for errors related to loan modifications, some categorized loans by the loan modification program (e.g., Home Affordable Modification Program (HAMP) or proprietary) and others categorized loans for reasons modifications were denied. Review parameters: For similar loan categories, some consultants anticipated conducting 100 percent review of all files in that category whereas other consultants planned to sample files. For example, some third-party consultants planned to review all rescinded foreclosures, whereas others proposed reviewing a sample of those loans. According to regulator staff, differences in the sampling plans reflected differences in the size and characteristics of the servicers’ loan portfolios and data systems. Regulator staff explained that they reviewed each proposed sampling plan to help ensure it met the parameters outlined in the guidance and would result in statistically valid results. However, according to OCC staff, they recognized that some consultants had not fully implemented the sampling approach as expected, and OCC is taking steps to address these differences for one of the servicers that is not subject to an amended consent order and must continue its review. Our analysis of the May 2011 sampling guidance provided by regulators found that the guidance was ambiguous about a key parameter that affected consultants’ sampling methodologies and contributed to differences in those methodologies. Specifically, in their May 2011 guidance, regulators did not indicate whether consultants should explicitly set an expected population error rate or provide consultants with direction on the factors they should consider when setting an appropriate expected population error rate to determine the size of the sample used for their analysis. GAO’s Financial Audit Manual and the standards of the American Institute of Certified Public Accountants (AICPA) have found that a key element of effective sampling is the determination of an appropriate sample size based on specified precision and reliability levels and an expected population error rate or frequency of errors. Regulators’ May 2011 guidance on sampling specified that consultants should use 3 percent precision and 95 percent reliability levels to determine their sample size. According to regulator staff, these precision and reliability levels were selected to provide a high-level of confidence in the sampling results. However, the guidance did not specify an expected population error rate for consultants to use in determining sample size. Generally, the expected population error rate, like precision and reliability levels, is determined based on professional judgment and includes consideration of factors such as results of prior reviews and knowledge about any potential risks in servicing and foreclosure processing errors. According to regulators, they expected consultants to find errors in their sampled files, because the consent orders that required the foreclosure review process arose out of assessments conducted by regulators that identified the potential for servicing and foreclosure-related errors. Similar to the precision and reliability levels, there is a relationship between the expected population error rate and sample size needed to attain a specific precision level (margin of error), where the size of the sample generally expands as the expected population error rate approaches 50 percent. When an expected population error rate is not specified, it can be interpreted as implying that the error rate is low or even zero—that is, that few or no errors are expected to be found in the sample regardless of whether the expected population error rate of zero is appropriate for the sample goals. OCC staff told us that their handbook on sampling methodologies—a reference their staff use for sampling and one that both regulators suggested consultants consider in designing their sampling approaches—generally assumes few or no errors will be found in a sample. Variations among the sample sizes used to analyze the various loan categories identified by consultants result from differences in the consultants’ expected population error rates and led consultants to use different triggers to determine when to conduct additional analysis of an error or errors found in a loan category. As shown in figure 1, Consultant A would conduct additional analysis of their sampled loan categories if one or more errors were found. In contrast, Consultant B would conduct additional analysis only when five or more errors were found in their sampled categories. Both of these approaches use the precision and reliability levels specified in the May 2011 guidance, with an expected maximum threshold of a 3-percent error rate, but relied on different sampling approaches and different expected population error rates to calculate sample size. Therefore, with the same number of errors identified through file reviews, some consultants would conduct additional analysis and some would not. According to regulator staff, when errors were found, they expected consultants to conduct additional sampling of the loan category or to review all of the loans in that category. OCC staff explained that after approving the original sampling methodologies, they recognized that their review of the plans had missed some aspects and that consultants were using various expected population error rates to calculate sample size. OCC staff told us that they were considering steps to try and address these differences for one of the servicers that had not joined the agreements to end the foreclosure review. As a result of this variation, the reviews could have produced inconsistent results for similarly situated borrowers, thereby potentially limiting achievement of one of the goals of the foreclosure review process. Our analysis also found that the May 2011 guidance on sampling did not include a discussion of regulators’ expectations for reporting on sampling, and variations among the sampling plans would have limited the types of information that regulators could report. For example, the guidance did not specify if regulators expected consultants, based on their sampling methodology, to be able to provide information on the error rate for the servicer’s whole population of eligible borrowers or for certain characteristics, such as high-risk loan categories (e.g., states). Our analysis found that 1 of the 14 reviews explicitly designed samples that would potentially allow consultants to generate reliable estimates of the error rate for certain states. Other consultants may have been able to report counts of errors by state, but some consultants sampled as few as one or five cases per state, and the counts would not allow regulators to determine whether reported errors were relatively frequent or infrequent in specific states. As a result, regulators could not have reported results by state because some reviews did not have enough cases, selected in a generalizable manner, from individual states to report statistically reliable estimates. According to regulators, reporting error rates was not a consideration in developing the sampling approach outlined in the guidance; rather, the sampling methodology was intended to find as many harmed borrowers as possible. However, it would be feasible to develop sampling approaches that could both produce error rates and identify characteristics of harmed borrowers. Similarly, differences in the loan categories analyzed by the third-party consultants would have limited regulators’ ability to easily aggregate results among the reviews or present comparable servicer-specific information. For example, differences in how consultants organized the loan modification category could have impeded regulators’ ability to aggregate the results of the loan modification analyses conducted by third-party consultants or to present equivalent information among the servicers to the public. In addition, these differences in the loan category definitions as well as other differences in the consultants’ sampling methodologies would have made it difficult for regulators to aggregate results to represent the full population of eligible loans. One consultant told us that they anticipated calculating an error rate based on the number of files reviewed compared to the number of files with errors. This method might have provided some information on errors, but could be subject to bias if the sample of loans reviewed were not representative of the population, regardless of whether the estimates from the samples were appropriately weighted for probability of selection or included appropriate confidence intervals to reflect sampling error. However, this information would not necessarily provide a statistically valid description of the extent of errors in the full population that regulators and other stakeholders would need to understand and assess the results. According to regulator staff, they expected consultants to continue reviewing files until as many harmed borrowers as possible were identified, a goal of the foreclosure review process. Regulator staff told us that if third-party consultants’ initial analyses of sampled loans identified errors within a loan category, consultants were expected to analyze the characteristics of loans with errors to identify any patterns and use this analysis as the basis for a second sampling phase or to review all the files in a loan category. Although not explicitly stated in the May 2011 guidance, according to OCC staff, additional analysis was warranted when the errors in the sampled loan category exceeded 3 percent with a 95 percent confidence level. Although the May 2011 guidance on sampling did not specify the characteristics to consider for this analysis, Federal Reserve staff told us that identifying the characteristics to include would require consultants to use their judgment and they anticipated it would include things like the loan category itself—such as an error related to a certain foreclosure attorney that might warrant additional review of other loans that had used the same attorney—or some other characteristic about the loan. According to a few consultants, they were considering characteristics such as the loan product or date of the loan modification solicitation or foreclosure sale, as potential characteristics. Regulators told us that based on the results of the consultants’ analyses of the initially sampled loans with errors, consultants were to develop a methodology—including conducting additional sampling or reviewing all files with those shared characteristics—to identify other loans that had similar characteristics that could have had similar errors to use as a basis for a second review phase. According to Federal Reserve staff, this process would have been iterative—where consultants would have found errors, analyzed those errors, resampled and then repeated the process until as many files with errors as possible had been found. Third-party consultants were expected to discuss the results of their initial file reviews, their analysis of error patterns, and their proposed approach for conducting additional file reviews with regulators prior to conducting additional reviews. Our analysis found that the regulators’ sampling approach did not include mechanisms to facilitate their oversight of the extent to which consultants would have reached as many harmed borrowers as possible. For example, the regulators’ sampling approach did not provide an objective method for regulators to use in determining if consultants had conducted sufficient reviews and could stop their review activities, except in those cases where there were few or no errors. As we described earlier, consultants were using different triggers of the number of errors to determine if additional analysis of loan categories was required. Without a mechanism for regulators to use in assessing the extent to which each consultant had found as many harmed borrowers as possible or to compare the review results among the consultants, assessing which consultants had done enough work to identify a sufficient portion of harmed borrowers and which consultants needed to conduct additional analysis would have been difficult for regulators. According to Federal Reserve staff, they anticipated that consultants would continue reviewing files until the sampling found no additional errors. However, unless a large majority of the population is examined for errors, reviewing files until sampling finds no more errors does not necessarily imply that all or most errors would be identified. OCC staff told us that at the time of the agreements that led to the amended consent orders, regulators were considering developing such mechanisms. Additional sampling activities—specifically, the use of baseline samples— could have provided regulators with a mechanism to use in monitoring the extent to which consultants had identified as many harmed borrowers as possible. The Office of Management and Budget (OMB) standards for statistical surveys state that where sampling is used, it should include protocols to monitor activities and provide information on the quality of the analyzed data. A baseline sample could have been used to establish an estimate of the number of harmed borrowers among the servicer’s full population as part of their sampling methodology and would have provided an objective and consistent measure for regulators to use to gauge if a third-party consultant had identified a sufficient portion of harmed borrowers through their reviews. Specifically, regulators could have compared a statistically valid estimate of the number of harmed borrowers in the servicer’s total population of eligible borrowers with the number of harmed borrowers the consultant identified through file reviews. Discrepancies between the estimated number of harmed borrowers and the number found in the review would have helped to indicate the extent to which there may have been additional harmed borrowers who had not been identified. Without this type of comparison, regulators did not have an objective measure to help determine when a consultant had completed sufficient review of the servicers’ files. In addition, the regulators’ sampling approach did not provide a clear mechanism for regulators to assess the extent to which consultants had identified the appropriate high- and low-risk loan categories to confirm that those categories were accurate and to signal if there were additional potential high-risk loan categories that had not been identified, but warranted additional sampling and review. As we noted earlier, the regulators’ sampling guidance suggested a number of high-risk loan categories consultants should consider in designing their sampling approach—such as rescinded foreclosure or foreclosures following loan modification—that were potentially associated with a higher likelihood of error and, depending on the servicer, warranted targeted sampling to verify the extent to which errors had occurred among the loans in those categories. However, there is evidence to suggest that other loan categories not included in the guidance, such as certain loan categories based on demographic characteristics may also be associated with higher likelihood of servicer errors. Without a mechanism to assess the accuracy and sufficiency of each consultant’s high- and low-risk loan categories, regulators would not have been able to assess the extent to which consultants were targeting the appropriate high- and low-risk categories to find as many harmed borrowers as possible. An estimate of the overall error rate for the population—a rate that could have been generated using information from the baseline sample—could have been compared with the error rates consultants found for high- and low-risk loan categories to confirm that those categories were accurate and to signal if there were additional potential high-risk loan categories that had not yet been identified. OMB has suggested that where different characteristics are being compared, agencies should conduct additional testing to help ensure appropriate statistical conclusions are derived from the data. In our prior work we have used this type of statistical testing to compare the performance (such as error rate) in a total population with the performance for subgroups—such as loan categories—to help distinguish among those groups. This can be part of conducting an overall risk assessment of the population to enable users to better focus on high- and low-risk areas. Without a mechanism to gauge the extent to which high-risk loan categories had been identified, regulators could have had difficulty assessing whether the consultants’ proposed activities for additional analysis were targeted at the appropriate high-risk loan categories to identify as many harmed borrowers as possible. The May 2011 sampling guidance did not include a requirement that consultants develop a baseline sample, but most consultants included one in their sampling methodology. According to regulator staff, they had not considered requiring consultants to include a baseline sample in their methodology because the purpose of sampling was to help find harmed borrowers by identifying concentrations of servicing and foreclosure- related errors as described in the consent orders, not to establish an error rate for the servicer. Our analysis found that most consultants included a baseline sample for the full review population in their sampling methodology. However, due to differences among the consultants’ baseline samples, the ability of regulators to use consultants’ baseline sampling results to assess consultants’ activities varies. For example, the samples were not designed to estimate actual population error rates and regulators’ guidance did not direct consultants to statistically test for differences in the servicer’s total eligible population and among high- and low-risk loan categories. Our analysis also found that the regulators’ approach to conducting repeated additional analyses to find as many harmed borrowers as possible potentially involved timeliness trade-offs. OMB’s standards for statistical analysis state that the sampling design should be appropriate to achieve the sampling goals; in this case regulator staff told us that the phase-one sampling was designed to identify patterns from among loans with errors to facilitate a second phase of analysis to find as many harmed borrowers as possible. Where the goal of sampling is to identify patterns from among loans with errors in a sample, using a larger sample size has the potential to provide more information to use in analyzing patterns and determining the appropriate next steps. For those consultants that used smaller sample sizes in their phase one analysis, for example, a sample size of 100 loans where one error would trigger additional analysis, finding patterns among the characteristics of those loans with errors may have been difficult in cases where there were few errors. As a result, these consultants may have had to conduct potentially time-consuming repeated sampling of certain loan categories where errors were found to find enough loans with errors to be able to analyze the findings and identify patterns that would have allowed them to conduct additional sampling or review. OCC staff told us that some consultants were considering using statistical modeling as part of their analysis of loans with errors to identify patterns from among those loans. However, in cases where the sample size is small and the number of loans with errors is small, the value of statistical modeling to identify patterns could be limited. According to Federal Reserve staff, due to concerns about the timeliness of the sampling process, regulators anticipated that they may have had to cut short the sampling process and provide remediation based on certain borrower profiles—that is, borrowers who shared characteristics associated with loans with a higher likelihood of error—without analyzing each of the loan files that shared that profile. In contrast, a larger phase one sample size designed with the goal of identifying patterns among errors may have provided consultants with more information on loans with errors to use as the basis for their additional analysis. In particular, this may have been valuable for loan categories with a relatively large number of loans—for example, our analysis found that in some cases a loan category had more than 10,000 loans and, for one review, as many as 50,000 loans—and having more information available to use when analyzing the results for these categories and determining next steps may have been helpful. Although a larger sample size does not guarantee a more timely review, a larger statistical sample may have provided additional opportunities for analysis even with a similar proportion of errors. In the case of the foreclosure review, where the goal was to use sampling to find as many harmed borrowers as possible by analyzing patterns from among loans with errors, larger sample sizes may have resulted in a more timely process and potentially may have identified harmed borrowers more quickly. According to regulators, they designed their sampling to have a low tolerance for errors so as to find as many harmed borrowers as possible. According to third-party consultants, regulators’ guidance did not address certain aspects of the foreclosure review and consultants had to use additional judgment and interpretation when applying certain guidance, increasing the risks of inconsistency among review results. Consultants also noted that regulators issued critical guidance throughout the review process and frequently updated guidance, which expanded the scope of the reviews and contributed to delays. For example, the Federal Reserve issued three clarifications of loan modification guidance and OCC provided seven responses to frequently asked questions on reviews of loan modifications. According to regulator staff, developing the foreclosure review process was intentionally iterative where they responded to the most immediate need and used their evolving knowledge to help refine the guidance. Consultants said that changes to guidance required them to develop new test questions, re-train reviewers, and redo file reviews. Although regulators issued numerous pieces of formal guidance and informally responded to specific questions about review procedures that examination teams or consultants raised, consultants said that some of the guidance issued was not specific, leaving room for their interpretation and potentially contributing to inconsistent interpretations. Guidance on fees: Guidance from regulators generally directed consultants to consider whether the fees servicers charged for actions such as property inspections or lawn care services were permissible under the terms of the loan, followed applicable state and federal law, and were reasonable and customary. However, regulators provided additional explanation in response to requests from consultants for clarification and guidance on defining the terms “reasonable” and “customary.” Consultants for 13 of the 14 reviews indicated that they used investor guidelines to determine if the fees charged were reasonable; however, one consultant told us consultants were using different versions of these guidelines. In some cases, consultants used additional methods to evaluate whether fees charged to borrowers were customary. For example, one consultant evaluated fees against a set of benchmarks created from multiple servicers’ actual fee charges, while others said they benchmarked only against the investor guidelines. Guidance on remediation: Although regulators issued guidance to consultants on how to determine the appropriate remediation for different financial harms, 4 out of the 13 remediation categories would have required consultants to make remediation determinations on a case-by-case basis, risking inconsistent treatment of borrowers. For example, consultants would have had to determine remediation on a case-by-case basis if they found that a servicer had initiated foreclosure or foreclosed on a borrower who was protected by federal bankruptcy law. Consultants reported taking a range of approaches to these case-by-case determinations, including awaiting further guidance, developing their own guidelines, or relying on recommendations from their third-party law firms. According to Federal Reserve staff, at the time of the agreements that led to the amended consent orders, regulators were considering options to provide additional guidance to consultants for work in these areas, including determining remediation amounts. Guidance on missing documentation: Guidance issued to consultants in March 2012 on how to determine whether a borrower suffered financial harm when key documents—such as documents that evidenced that foreclosure actions were taken or loan modification application documents—were missing indicated that consultants should treat those instances as errors on the part of servicers. However, this guidance noted that consultants should defer decisions on how to determine borrower remediation until regulators provided further guidance. In the absence of additional guidance, consultants indicated that they took a variety of approaches, including waiting to make remediation decisions, making preliminary considerations of whether the error might have caused financial harm, or working with their third-party law firm to review any applicable legal precedents. According to OCC staff, at the time of the agreements that led to the amended consent orders, they were planning to issue further guidance to consultants on remediation for borrowers with missing documents and they had informally provided additional direction about treatment of files with missing documents during their regular meetings with consultants. According to third-party consultants, regulators missed key opportunities to increase the likelihood of consistent outcomes for borrowers by not requiring development of common criteria or reference materials that served as the basis for the foreclosure review process. Third-party consultants and their respective law firms we interviewed told us that they each developed their own test questions used by their file reviewers to determine whether any errors or financial harm occurred. Consultants developed the test questions based on analyses of state foreclosure laws, loan modification guidelines, and bank policies, among other references. According to OCC staff, the state law references were fairly straightforward and they had confidence that the third-party consultants and law firms would provide fairly consistent interpretations. However, according to third-party consultants and law firms we interviewed, compiling these references and using them to develop review questions was challenging and time consuming and, in some cases, required judgment or interpretation of the laws or guidelines. For example, they noted that certain areas of relevant state law were unsettled and continued to evolve as courts issued decisions. In addition, representatives of law firms involved in the reviews told us that each firm developed its own list of state foreclosure requirements and often came to interpretations different from those of other law firms involved with the foreclosure review. Consultants also indicated that some law firms had different interpretations of which laws were applicable. According to OCC staff, the scope of the reviews was limited to federal and state laws, but several consultants reported that they reviewed servicers’ compliance with certain county- or court-level requirements, whereas other consultants said they generally did not include these requirements. Although OCC and Federal Reserve staff told us that law firms were selected for their independence and capacity to interpret these documents and make these types of decisions, with multiple law firms developing their own interpretations of the applicable laws and requirements, consultants may have based their test questions on different interpretations of laws, which could have hindered regulators’ ability to achieve one of their goals for the foreclosure review, similar treatment for similarly situated borrowers. Our analysis indicates that regulators missed another opportunity to standardize reference materials in the area of loss mitigation and loan modification requirements. Consultants noted challenges in compiling relevant loan modification program guidelines. For example, according to consultants, they had to compile requirements for multiple loan modification programs and representatives of one consultant we interviewed said that they had to compile requirements of 40 different loan modification programs that the servicer used during the 2009 to 2010 period. In addition to numerous different programs, the guidelines for a single program may have changed multiple times. One consultant noted that they had to track and apply 28 program changes the Department of the Treasury (Treasury) made to HAMP between 2009 and 2010. Our prior work on HAMP identified instances of servicers interpreting HAMP guidelines inconsistently, and a consumer advocacy group report noted similar challenges with servicers implementing HAMP according to the guidelines. In addition, several consultants from whom we obtained information told us that they had to make some judgment calls when interpreting regulatory guidance and program guidelines. For example, one consultant noted that HAMP guidelines were unclear in critical respects, particularly regarding the nature and extent of servicers’ obligations to notify borrowers about steps in the HAMP process. OCC staff said they were aware of Treasury’s HAMP guidance and other servicing guidelines and made an effort to make their guidance to consultants consistent with these materials. OCC and Federal Reserve staff also stated that certain staff members had general discussions with Treasury staff to understand the HAMP guidelines. Consultants we interviewed told us that they largely relied on their internal subject-matter experts to interpret the relevant guidelines and did not consult with program experts, such as Treasury staff who designed and developed HAMP. As a result, third-party consultants and law firms may have applied different interpretations of the legal or program requirements for the same programs to the reviews, and the use of different reference materials could have reduced the likelihood of achieving the goal of treating similarly situated borrowers consistently. Regulators took steps to monitor potential inconsistencies among the reviews, but these steps were limited and likely would have resulted in delays in providing remediation to borrowers. First, according to regulators, they closely monitored weekly reports provided by consultants to identify any differences in their progress that may have indicated some inconsistency in the foreclosure review processes. These initial reports included information on the number of mailings; requests for review; and high-level counts of file reviews started, in process, and completed and the number of files with borrower harm, but they did not include information on the specific types of errors identified in the reviews or the test criteria used to review files. Therefore, the usefulness of the reports for identifying inconsistencies was limited. According to OCC staff, they planned to begin requiring consultants to report on additional information, such as the types of errors associated with financial harm found in the reviews and proposed remediation amounts, which would have helped identify inconsistencies among reviews. Second, OCC and Federal Reserve staff said that they would identify potential inconsistencies among the reviews by having staff from one examination team assist with another team’s oversight of the foreclosure review, but these rotations among the staff were not systematically organized and did not include rotations across regulators. Further, some examination team members we interviewed said that they had participated in one or two reviews for other servicers, but examiners overseeing reviews at a larger servicer noted that they were unable to participate in multiple reviews because of time constraints. In addition, OCC and the Federal Reserve did not provide the examination team members conducting these rotations with guidance on the types of issues to consider in assessing inconsistencies. This unsystematic approach limited the extent to which regulators would have been able to identify trends or inconsistencies. Third, according to regulators, they reviewed some test questions but did not compare them across reviews to identify inconsistencies. OCC staff acknowledged that inconsistencies were inherent in the foreclosure review because of the large number of actors and decision points and the subjective nature of some of the decisions consultants had to make. Similarly, Federal Reserve staff told us that inconsistencies among reviews were inevitable due to differences among the servicers’ policies, procedures, and systems, including their loan modification and loss mitigation programs. Regulator staff said they had planned to conduct assessments of the extent of inconsistencies affecting the outcomes for borrowers across the reviews after the reviews and recommendations for remediation were completed. However, conducting such an assessment after the reviews were completed could have resulted in delays in remediation because third-party consultants may have needed to change their file review questions and redo file reviews if regulators identified inconsistencies. In addition, regulators would have had to wait until all consultants had completed their reviews to conduct such an assessment. Consultants reported that they had estimated completing their reviews in different time periods and an OCC official estimated that the reviews would not have been completed until 2014. Our analysis of the foreclosure review process identified challenges in regulators’ planning for the foreclosure review. These challenges underscored the importance of the following three aspects of planning and monitoring: (1) identifying the type and amount of information to report as final results during the design of data analysis; (2) consulting with stakeholders; and (3) assessing how well the reviews were tracking their goals of identifying as many harmed borrowers as possible, achieving consistent reviews for borrowers, and helping to restore public confidence in the mortgage market. As described earlier, variations among the 14 sampling methodologies used by third-party consultants for analysis of loan files limited the types of information that regulators would have been able to report. According to regulator staff, they did not want to make final decisions on the types of information they may have needed for public reporting before they had seen the preliminary results from the file reviews. OMB has found that in designing data analysis activities, including sampling, agencies should consider the types of data they need to collect to be able to report useful information on the results of their activities to the intended audience. The resulting sample design should have these data output requirements built into the structure. GAO’s internal control standards state that producing reliable and relevant data is important for oversight and management, including oversight provided by Congress. In addition, our prior work has found that public reporting of results can be important for strengthening public confidence in a process, and the data analysis strategy should be designed to include information that can be reported publicly. Providing useful and relevant public reporting of the review results also was a key element in renewing public confidence in the mortgage servicing market, a goal of the foreclosure review process. However, regulators were limited in what they would have been able to report because they did not plan for reporting in the design of the reviews. As described earlier, the scope of the foreclosure review was broad and regulators experienced challenges in issuing guidance on the wide variety of complex issues covered by the reviews, resulting in delays in completing file reviews and potentially contributing to inconsistencies in the file review process. Regulators may have been able to better define the scope of activities and issue more complete guidance prior to commencing the foreclosure review process, thereby potentially reducing the number of revisions to the scope or guidance, by consulting with organizations directly responsible for or familiar with particular aspects of the review before initiating the foreclosure review process. For example, although regulators consulted with Treasury staff for their technical expertise on the HAMP requirements during their development of the loan modification and loss mitigation guidance, regulators did not have the benefit of additional consultations with Treasury compliance officials and discussions with other agencies responsible for overseeing federal loan modification and loss mitigation programs to help them more clearly define the programs covered by the consent order requirements and the elements to consider in assessing servicers’ evaluation of loan modification and loss mitigation activities—areas where regulators issued clarifying guidance to third-party consultants. Regulators issued additional guidance to clarify that the review of HAMP and proprietary loan modification programs should include programs overseen by the U.S. Department of Housing and Urban Development, U.S. Department of Agriculture, and U.S. Department of Veterans Affairs. In addition, regulators did not consult with community groups, such as national organizations representing housing counselors that have worked with individual borrowers on their loan modification and loss mitigation applications. These consultations might have provided input on challenges specific servicers and borrowers experienced with the range of loss mitigation and loan modification activities which could have assisted in identifying high-risk loan categories or program elements to consider. For example, our prior work surveying housing counselors found that while assisting borrowers with HAMP applications, counselors experienced challenges with servicers, including missing documentation, lengthy decision-making processes, and miscalculations of borrowers’ incomes. Our prior work that establishes generally accepted project planning practices identified consulting with stakeholders as one of seven generally accepted practices. In addition, our internal control standards have found that consulting with external stakeholders can have a significant impact on the achievement of goals. In contrast to the process used to develop the loan modification and loss mitigation guidance, OCC and the Federal Reserve consulted with consumer groups while developing the remediation framework. In addition, they consulted with the U.S. Department of Justice in developing the foreclosure review guidelines related to SCRA. Although this consultation occurred later in the process after third-party consultants had begun SCRA reviews, one consultant cited the guidance that resulted from the consultations as evidence of a strong practice that helped promote consistent results among the reviews. As we described earlier, regulators’ sampling approach did not include mechanisms that would allow them to objectively measure the extent to which consultants were on track to identify as many harmed borrowers as possible. Similarly, as we previously described, regulators had a limited process in place to identify inconsistencies among consultants’ file review processes that could have affected results for borrowers. OCC and Federal Reserve staff told us that they were aware of some areas where there may have been inconsistencies, but according to Federal Reserve staff these areas would not have led to significant differences. However, in the absence of mechanisms to systematically monitor consistency, regulators did not have the information to verify their understanding or identify areas of the review with an increased likelihood of inconsistency. Our prior work has identified using intermediate activities or measures to assess progress toward intended results as an effective management practice to understand the extent to which activities are on track to reach stated goals. We have also identified practices that can help agencies successfully implement the Government Performance and Results Act and related results-oriented management initiatives, such as establishing activities during program planning and design to monitor performance toward these goals and using intermediate activities to analyze the gap between where the performance is and where it needs to be to achieve desired outcomes. We found that such activities can help management target areas in need of improvement and select appropriate methodologies to realize that improvement. In the absence of systematic processes to monitor the extent to which the foreclosure review was progressing toward its goals, regulators did not have an early warning mechanism to help identify problem areas where interventions, such as the issuance of clarifying guidance or other support, might have helped reorient activities and address concerns. Regulators publicly released information on the foreclosure review process beyond what is typically disclosed in connection with a consent order, including engagement letters between servicers and consultants and some guidance provided to the consultants. By law, federal banking regulators must disclose any formal enforcement actions entered into under the Federal Deposit Insurance Act. On a case-by-case basis, banking regulators may consider the release of information beyond the mandatory disclosures. In an effort to promote transparency in the foreclosure review process, OCC and the Federal Reserve publicly disclosed some information related to the April 2011 consent orders. For example, in November 2011, OCC released redacted engagement letters between the servicers under its jurisdiction and the consultants contracted to conduct the foreclosure review. With the exception of one servicer, the Federal Reserve released by February 2012 redacted engagement letters for servicers under its jurisdiction. OCC and the Federal Reserve also released the remediation framework for consultants to use that provided examples of situations in which compensation or other remediation is required for financial injury due to servicer errors, misrepresentations, or other deficiencies. Despite these disclosures, some stakeholders perceived gaps in key information about how the file reviews were conducted. Regulators released documents, such as the redacted engagement letters and remediation framework, which generally described the design and intended outcomes of the foreclosure review, but they did not disclose the more detailed guidance and tests consultants relied on to perform their reviews. As previously discussed, third-party consultants developed thousands of test questions to determine error and harm, and regulators issued a number of guidance documents to promote consistency among the reviews and clarify issues raised by consultants, such as how consultants were to construct their sample populations and how to address issues related to borrowers covered by SCRA. Although regulators released their remediation framework and provided answers to frequently asked questions on remediation categories and calculations, they did not release any additional guidance documents nor did they publicly disclose consultants’ test questions, which according to OCC, contained proprietary and supervisory information. To increase the transparency and credibility of the foreclosure review for borrowers, policy makers, and the public, among other stakeholders, consumer groups recommended that regulators release such information. According to consumer groups, without such information, the public would have questions and doubts about how the reviews were executed. OCC and the Federal Reserve staff said that they considered releasing additional guidance to the public, but both regulators refrained from doing so because of concerns that releasing detailed information risked disclosure of confidential or proprietary information. Moreover, test questions developed by consultants were numerous and complex, and Federal Reserve staff stated that review processes were too dissimilar to provide a comprehensive summary. Borrowers who requested reviews under the foreclosure review process initially received limited information about the status of their individual file review. Borrowers received a letter acknowledging their request was received, but some did not receive updates until almost a year after the outreach program was launched, when they received a letter informing them of the continuing nature of the review. In letters to OCC and the Federal Reserve, consumer groups indicated that these borrowers were frustrated by the lack of information on their particular file review. Eligible borrowers could submit requests as early as November 2011. According to OCC staff, borrowers who submitted an accepted request-for-review through June 30, 2012 received a status update letter in September 2012. OCC staff said that the letters communicated to borrowers that their requests were being reviewed but that the results of the review might not be available for several more months. They said that the letters also provided a brief summary of the foreclosure review process, an Internet link to the interagency remediation framework, and notice of other help available through nonprofit organizations approved by the Department of Housing and Urban Development. Regulators indicated that additional status letters would be sent to borrowers with outstanding requests-for- review. Before the regulators halted the foreclosure review, draft letters to be sent to borrowers on the results of their file reviews were in development. However, regulators were still uncertain about specific information they would require be shared with both borrowers who would receive remediation and those who would not. Regulators have acknowledged the importance of transparency, but when announcing the agreements that led to the amended consent orders, they had not yet determined what information to convey beyond that which was included in their press releases and public websites, nor had they determined whether additional information would be provided to borrowers who submitted a request-for-review. During the foreclosure review process, OCC released two interim reports that provide the public with information on the organization and conduct of the file review process and preliminary results, such as the number of requests-for-review received, for institutions it supervises. The OCC reports—issued in November 2011 and June 2012—summarized the status of actions taken to correct deficiencies in mortgage servicing and foreclosure processing identified in the April 2011 consent orders, including activities related to the foreclosure review. For example, the June 2012 report included the number of files selected for review, the number of requested reviews, and the number of reviews completed, among other items. These reports, according to OCC, were intended to build transparency in the process. The Federal Reserve did not issue interim reports on the foreclosure review process for institutions it supervised. According to Federal Reserve staff, they did not do so because they determined that their public release of servicers’ action plans provided sufficient information about how servicers were addressing the requirements of the consent orders and their public release of servicers’ engagement letters provided sufficient information about how the foreclosure review would be conducted. Prior to the announcement of the agreements that led to the amended consent orders and ended the foreclosure review for most servicers, OCC staff told us they had planned to release a final report on the results of the foreclosure review. After the agreements were announced, Federal Reserve staff indicated they expected to publish additional relevant information related to the foreclosure review and the agreements. However, as of February 2013, regulators had not decided what specific information will be made available on the work conducted under the foreclosure review prior to the agreements. While OCC and the Federal Reserve acknowledged the importance of transparency in the foreclosure review process, the absence of timely and useful communications at certain stages of the process—for individual borrowers as well as the general public—hindered transparency and undermined public confidence in the processes and results. In a December 2011 letter to consultants, OCC noted the importance of public confidence in the foreclosure review process. According to the Federal Reserve, the agency endorsed OCC’s letter. However, as previously discussed, regulators did not publicly release detailed information that described how consultants were to determine errors and remediation, and borrowers and the general public received limited information about the status of the reviews. As a result, consumer groups raised concerns about the level of transparency of the foreclosure review process and indicated that the absence of public information undermined credibility and public confidence in the process. Our internal control standards state the importance of relevant, reliable, and timely communications within an organization as well as with external stakeholders. As illustrated in Treasury’s implementation of the Troubled Asset Relief Program (TARP), external communications can include posting information on its website and regular, public reporting. Our previous work on TARP described the importance of public reporting as a means to improve transparency and address potential questions of whether similarly situated borrowers are being treated fairly. For example, Treasury periodically issued public reports on the progress and performance of its TARP housing programs. These reports have provided information to a range of stakeholders, including Congress and the general public. Similarly, consumer groups recommended that regulators provide regular, public reports on the progress and findings of the foreclosure review to increase transparency. Consumer groups also suggested that regulators provide additional information to address transparency-related issues for borrowers eligible for remediation through the foreclosure review. For example, they indicated that borrowers should have had access to information about the status of the review of their file and receive a thorough explanation of how decisions were reached. In addition, they recommended that the review guidelines issued by the regulators be made public, similar to Treasury’s release of guidance for HAMP. According to consumer groups, these actions, among others, could have increased the public’s understanding of the reviews and allowed borrowers to ensure their files were properly reviewed. The foreclosure review, unlike TARP and TARP-funded programs such as HAMP, was not a government program. However, as OCC described, it was part of a larger set of government-directed actions in response to the housing and mortgage crises. More publicly disclosed information about processes and regular reporting about the status of the reviews would have increased transparency and thereby public confidence in the reviews, given that one of the goals regulators articulated for the foreclosure review was to restore public confidence in mortgage markets. The foreclosure review revealed three key lessons that could help inform regulators’ implementation of the amended consent orders: (1) designing project features during the initial stages of the process to influence the efficiency of file reviews, (2) monitoring progress to better ensure the goal of achieving intended results, and (3) promoting transparency to enhance public confidence. These key lessons on planning and implementation could help contribute to an effective process for distributing direct payments and other assistance as prescribed by the amended consent orders between servicers and regulators. In addition, these lessons could inform the foreclosure review process that is continuing for the servicers that did not reach agreements with regulators. The foreclosure review experience suggests that a planning process to determine key project features, such as guidance and necessary data elements, for activities conducted under the amended consent orders could lessen the risk of changes to the planned activities, future delays, or rework. Our work on designing evaluations, including financial audits, has found that systematic and comprehensive planning enhances the quality, credibility, and usefulness of the results and contributes to a more effective use of time and resources. We found that one of the first steps in designing a review should be to define the purpose and scope of the review, including defining what data will be collected and what comparisons will be made. In addition, evaluation questions should be clear and specific, and should use terms that can be defined and measured so that the purpose and scope are readily understood and feasible. Our prior work establishing a lessons-learned process also has found that assessing and using lessons learned from previous experience can provide a powerful method of ensuring that beneficial information is factored into the planning and work processes of future activities. Key practices of assessing lessons learned include collecting and analyzing information on prior activities and applying that information to future activities. As the foreclosure review experience suggests, some implementation challenges were inevitable due to the unprecedented nature of the review. Nevertheless, the broad and expanding scope of the reviews and delays in defining key concepts could have been mitigated by more advanced planning from regulators, resulting in more efficient and effective reviews. The April 2011 consent orders provide a general description of the scope of the file reviews consultants were to conduct. According to third-party consultants, these parameters resulted in a broad review scope that generally covered all instances of noncompliance with applicable law and servicing guidelines as they related to the post-default residential mortgage loan borrower experience, including instances of noncompliance that resulted in financial harm to borrowers requiring remediation. Regulators may have missed opportunities to potentially narrow and refine the project scope—for example, through earlier definition of a harmed borrower or agreement on errors not resulting in remediation that may not have warranted additional review. Changes to guidance also expanded the scope of the reviews. For example, as we noted previously, changes to loan modification guidance contributed to an expanded scope for those reviews and delays in completing the work, including, in some cases, redoing file reviews. According to regulator staff, developing the file review process was iterative; they learned as the reviews progressed, and regulators used that knowledge to help refine the review process. In our work on designing evaluations and audits, we recognize that a review process can be iterative and that the scope and activities of the review may change as work progresses and data limitations or new information arise. Nevertheless, conducting a planning process that involves all stakeholders provides an opportunity to examine preliminary information and pilot-test processes and procedures to help further define the scope of potential activities and hedge against the risk of future changes. In addition, assessing lessons learned by using project critiques and discussions with key participants and stakeholders—such as local examination team staff, third-party consultants and law firms, and external groups—could identify the root causes of strengths and weaknesses of the foreclosure review that could apply to the amended consent order activities. According to regulator staff, they are meeting with examination staff and third-party consultants to discuss challenges with the servicers’ data that may make it difficult for servicers to determine borrowers’ direct payment amounts under the amended consent orders. OCC staff also said that they had discussions with other federal agencies and consumer advocacy groups before announcing the agreements that led to the amended consent orders. As regulators prepare to implement the amended consent orders and compensate borrowers, they may encounter delays and inconsistencies similar to those associated with the foreclosure review if they miss opportunities to make key project planning decisions. Regulators told us that they anticipated borrowers would be contacted by the end of March 2013 and that they also expected the issuance of checks to begin in April. However, regulators still need to make some key decisions about these activities in order to meet their goal of beginning borrower payments in April 2013, and clear guidance in several areas will facilitate the process. According to regulator staff, prior to contacting borrowers each servicer will assign borrowers to direct payment categories. OCC staff said that the servicers were instructed to use objective data and definitions and guidance from regulators in placing borrowers in the categories and that the examination teams would use the guidance provided to servicers to conduct their validation process. Regulator staff said that they agreed that clear guidance on the categorization and file review process and validation by examination teams are tools that will help the servicers and examination teams navigate this complex process. In most cases, servicers, with regulators’ approval, have engaged the third-party consultants to review borrowers’ files in two categories (SCRA and foreclosed borrowers who were not in default) to determine whether borrowers experienced those specific types of harm. According to one third-party consultant, at the time of the agreements that led to the amended consent orders, consultants were waiting on additional guidance from regulators to complete aspects of these reviews. In addition, regulators have not yet determined payment amounts for the different categories of borrowers. Regulator staff said that they could not make these decisions until servicers had completed their categorization process and regulators knew the number of borrowers in each category to allow them to divide up the total payment amount among the borrowers. Regulators also have not determined what to do with any funds that might be left because borrowers refused payments, did not cash their checks, or could not be located. According to regulators, any remaining funds will not be returned to the servicers. Further, regulators told us that the agreements specify that over the next 2 years servicers are required to take loss mitigation and foreclosure prevention actions for which each servicer will earn credit toward fulfilling a specified obligation. These activities will provide assistance to borrowers covered under the consent orders and other borrowers. Regulator staff said they provided a list of eligible activities to servicers and advised them to prioritize borrowers covered under the consent orders for assistance. However, according to OCC staff, they did not provide servicers with criteria for identifying borrowers to whom they will offer mortgage assistance, which would help ensure that eligible borrowers have a consistent opportunity to be considered for these funds. Without assessing past lessons learned and making decisions on key features of the amended consent order activities in advance, regulators risk having to implement changes in the planned activities or publicly announced timelines, which could decrease the efficiency of the process to distribute direct payments and other assistance. The foreclosure review experience suggests that regulators’ process for monitoring third-party consultant, servicer, and examination team activities, including holding regular meetings and reviewing weekly progress reports, could provide a useful model for monitoring activities under the amended consent orders. According to regulators, examination teams will play a critical role in overseeing servicer activities under the amended consent orders, including testing and validating the results of the servicers’ categorization of borrowers. Regularly collecting and reviewing information on the activities and approaches of the examination teams could provide an opportunity to identify challenges and take steps to address them as the activities progress. Similarly, instituting a process to monitor the progress of the servicers’ loan categorization and track the payment administrator’s distribution of payments could help regulators assess the extent to which they are on target to reach their goal of providing notification to borrowers about their payment by the end of March 2013. OCC staff said that they continue to hold regular meetings with the servicers and examination teams to answer questions and share ideas. In addition, OCC staff told us that they intend to conduct site visits with each servicer, review servicers’ draft categorizations, and implement a reporting mechanism to oversee servicers’ activities under the amended consent orders. Regulators’ experience with the foreclosure review suggests that identifying comparative oversight mechanisms to promote consistency could help achieve consistent results among key actors. As discussed earlier, regulators had limited and unsystematic centralized control mechanisms to monitor consistency among the foreclosure review processes and did not have the information to assess the implications of any differences. According to regulators, achieving consistent results for borrowers, so that similarly situated borrowers receive similar payment amounts, is a goal of the amended consent orders, as it was of the foreclosure review process. OCC staff stated that the direct payments provided under the amended consent orders will likely be more consistent than what would have occurred under the foreclosure review because servicers are using a standard framework and objective criteria to categorize borrowers and all borrowers in a particular category will receive the same payment amount. According to regulator staff, the categorization instructions provided to servicers, regular meetings with servicers and examination teams, site visits, and examination team verification processes will provide opportunities to review and discuss the results of each servicer’s categorization of borrowers. However, whether there is a similar process that will compare results across OCC and Federal Reserve supervised servicers is unclear. Furthermore, to what extent regulators will assess the implications of any inconsistencies among the reviews that consultants are continuing to conduct for the servicers that did not sign agreements with regulators is unknown. Applying lessons from the foreclosure review process and our internal control standards to the amended consent order activities would suggest that mechanisms to centrally promote consistency and monitor agreement activities could help achieve consistent results for borrowers. GAO’s internal control standards state that agencies should take steps to comprehensively identify and analyze program operations to determine if risks exist to achieving goals—such as risks to the regulators’ goal of providing similar results for similarly situated borrowers. In our prior work, we found that using a horizontal review mechanism is an option to help mitigate risks of inconsistent results for activities conducted by multiple entities. For example, comparing servicer decision-making processes, including any criteria used by the servicers to categorize borrowers, could identify any potential differences and the extent to which these differences may result in different direct payment decisions for similarly situated borrowers. Similarly, mechanisms to provide clear and specific guidance—such as guidance on testing and validation of servicer activities—for local examination teams to use in their oversight of individual servicer activities could help regulators to more easily monitor and compare servicer activities and the results for borrowers among the reviews. Without using mechanisms to centrally monitor the consistency of servicers’ activities to categorize borrowers, regulators may risk delays in providing direct payments to borrowers and inconsistent results. In addition, without monitoring potential inconsistencies in the foreclosure reviews that are continuing for servicers that are not party to the amended consent orders, regulators will not have the information to assess whether those servicers’ borrowers are being treated consistently. Lessons from foreclosure review activities conducted to date suggest that developing and implementing an effective communication strategy that includes public reporting goals could enhance the transparency of the activities under the amended consent orders. GAO’s internal control standards state the importance of relevant, reliable, and timely communications within an organization as well as with external stakeholders. As a means to strengthen communication with external stakeholders and improve transparency and accountability, our work on TARP has underscored the importance of a communication strategy. Moreover, our prior work on organizational transformation demonstrates that public and private-sector leaders view establishing a communication strategy as especially crucial in the public sector, where policymaking and program management demand transparency and stakeholders are concerned not only with what results are to be achieved, but also with which processes are to be used to achieve those results. In addition, as previously discussed, public reporting is also a mechanism for external communication that can enhance transparency. Experiences with current government initiatives that are aimed at assisting struggling homeowners and involve institutions and mortgage-related issues similar to those of the foreclosure review also highlight the benefits of regular performance reporting. Specifically, periodic reports on the performance of and participation in TARP programs and scheduled reports on servicers’ compliance with requirements of the National Mortgage Settlement are intended to promote transparency and build public confidence. Because the foreclosure review and the subsequent activities under the amended consent orders—like TARP and the National Mortgage Settlement—are part of the larger governmental response to the housing and mortgage crises, a communication strategy which incorporates plans for periodic public reporting may enhance transparency in the distribution of direct payments and other assistance and help restore confidence in the mortgage market. Regulators announced the agreements that led to the amended consent orders without a clear communication strategy. As such, what information will be provided to individual borrowers and the general public about processes, progress, and results of activities under the amended consent orders is unclear. Although OCC and the Federal Reserve have provided some information on the amended consent orders and have plans to release additional information, regulators have not made key decisions on communicating directly with individual borrowers and the extent to which they will report on activities related to the amended consent orders and continuing foreclosure reviews. Regulators provided limited information on the amended consent orders through press releases and updates on their websites, among other ways. For example, OCC and the Federal Reserve issued joint press releases announcing the agreements and related amended consent orders, and OCC and the Federal Reserve posted answers to frequently asked questions on their websites on the agreements that led to the amended consent orders. In addition, regulators plan to release information about the distribution plan after payment amounts are determined, but staff did not describe plans to release additional information about the procedures servicers are using to categorize borrowers. As of January 2013, OCC staff said that they planned to release two public reports, one in April 2013 to discuss the direct payment process, and one in the summer of 2013 to discuss the foreclosure review results of servicers not covered under the amended consent orders. OCC staff told us, however, that they have not decided on the specific content of these reports. As of February 2013, the Federal Reserve also plans to issue public reports about servicers’ activities under the amended consent orders and the results of the foreclosure reviews at servicers not subject to the amended consent orders. Regulators had not decided what, if any, information will be made available on the results of the work conducted under the foreclosure review prior to the agreements. Further, OCC and Federal Reserve staff told us they had not made decisions about the form and specific content of communications, if any, directly to individual borrowers. While the amended consent orders terminate the foreclosure review for most of the servicers, transparency of past and current efforts continues to be important to stakeholders, including Congress and consumer groups. In particular, members of Congress have expressed concerns about the lack of public reporting on the foreclosure review and a lack of information about how the amounts of payments and other assistance in the amended consent orders were determined. In addition, consumer groups expressed concerns about transparency and urged regulators to take additional steps to increase transparency and public confidence in the implementation of the amended consent orders, including robust data collection and reporting. In the absence of a clear communication strategy to direct external communications, including public reporting and direct communication with individual borrowers, regulators face risks to transparency and public confidence similar to those experienced in the foreclosure review process. The foreclosure review was intended to identify as many harmed borrowers as possible, ensure that similarly situated borrowers received similar results, and help restore public confidence in the mortgage market. Ultimately, the complexity of the foreclosure reviews and limitations in regulators’ guidance and monitoring of the foreclosure review challenged their ability to achieve the stated goals. OCC and the Federal Reserve took a number of steps to foster consistency of this unprecedented review, including issuance of nearly identical consent orders and joint issuance of guidance documents for third-party consultants. However, other efforts related to guidance and monitoring may have exacerbated the challenges and complexities inherent in the process. In particular, existing guidance on sampling was ambiguous, leading to inconsistent sampling methodologies used by consultants, and did not include key oversight mechanisms to facilitate assessment of the extent to which consultants had identified as many harmed borrowers as possible. In addition, regulators’ limited monitoring of consistency of the consultants’ sampling methodologies and review processes increased risks that similarly situated borrowers would receive different results. As a result, the regulators risked not achieving the intended goals of identifying as many harmed borrowers as possible and treating similarly situated borrowers similarly, and this remains a challenge for the servicers continuing the foreclosure review. Our prior work has identified practices, such as assessing progress toward goals and designing such monitoring during the planning stage of a project, as effective management practices. In addition, OMB has found that in planning data analysis activities, such as sampling, agencies should take necessary steps to ensure that they have collected the appropriate data from which to draw conclusions. Assessing the review processes of the continuing reviews for consistency and implementing a mechanism to assess the sufficiency of additional sampling activities could help mitigate risk of similarly situated borrowers receiving different results and facilitate oversight of the extent to which consultants have reached as many harmed borrowers as possible. Although the regulators have terminated activity related to the foreclosure review for the servicers with amended consent orders, the foreclosure review process offers an opportunity for the regulators to leverage this experience to help ensure that similar difficulties are better addressed in future efforts. In general, identifying, assessing, and using lessons learned can help ensure that beneficial information is factored into the work processes of future activities, among other things. Therefore, consideration of lessons from the foreclosure review activities, such as advance design of data analysis activities and assessment of inconsistencies in the file review processes, is one way to help regulators improve the clarity of their guidance and ensure effective monitoring of progress toward results for the activities under the amended consent orders and the three remaining servicers still subject to the foreclosure review requirement that will cover 450,000 eligible borrowers. Specific activities include the following. Sound planning and additional oversight mechanisms can enhance project design and help ensure achievement of program goals. Our internal control standards on monitoring, risk assessments, and consultations with stakeholders, as well as generally accepted project management practices for agencies to use in implementing the Government and Performance Results Act and related management initiatives, emphasize the importance of planning, monitoring, and assessing lessons learned. Assessing the strengths and weaknesses of the foreclosure review process by using project critiques and discussions with key participants and stakeholders—such as local examination team staff, third-party consultants, law firms, and external groups—could help ensure better design and more efficient implementation of activities under the amended consent orders. As such, consideration of lessons from the foreclosure review processes would enhance the design, implementation, and oversight of the activities under the amended consent orders. The foreclosure review activities to date also highlight the importance of effective communication. We found that limited communication with individual borrowers and the general public hindered transparency and public confidence in the reviews. Further, regulators announced agreements that led to the amended consent orders and ended the foreclosure review with 11 servicers without a clear communication strategy, and the information that will be provided to borrowers and the general public remains unclear. Our internal control standards state the importance of external communication, and our key practices in implementing transformations that were identified by public and private sector leaders underscore the importance of a communication strategy. As such, the development and implementation of an effective communication strategy could enhance regulators’ efforts to ensure transparency and public confidence in the results of the foreclosure review as well as processes to implement the activities under the amended consent orders. We are making three recommendations to the regulators: (1) To better ensure that the goals of the foreclosure review are realized for servicers that are not subject to amended consent orders, we recommend that the Comptroller of the Currency and the Chairman of the Board of Governors of the Federal Reserve System, as appropriate, improve oversight of sampling methodologies and mechanisms to centrally monitor consistency, such as assessment of the implications of inconsistencies on remediation results for borrowers in the remaining foreclosure reviews. (2) To better ensure that the goals of the amended consent orders related to the distribution of direct payments and other assistance are realized, we recommend that the Comptroller of the Currency and the Chairman of the Board of Governors of the Federal Reserve System identify and apply lessons from the foreclosure review process, such as enhancing planning and monitoring activities to achieve goals, as they develop and implement the activities under the amended consent orders. (3) To better ensure transparency and public confidence in the activities under the amended consent orders and results of the continuing foreclosure reviews, we recommend that the Comptroller of the Currency and the Chairman of the Board of Governors of the Federal Reserve System develop and implement a communication strategy to regularly inform borrowers and the public about the processes, status, and results of the activities under the amended consent orders and continuing foreclosure reviews. We requested comments on a draft of this report from OCC and the Federal Reserve. OCC and the Federal Reserve provided written comments that are presented in appendixes II and III, respectively. The regulators also provided technical comments, which we have incorporated into the report, as appropriate. In commenting on the report, OCC and the Federal Reserve both identified actions that they have taken or planned to implement the recommendations. Specifically, OCC stated that it plans to continue to ensure that its sampling guidance is used in the continuing reviews and that the agency plans to monitor for consistency. As we discussed in the report, developing and using objective measures to monitor and assess consistency among the continuing reviews is also important. With respect to the second recommendation, OCC stated that it plans to apply lessons learned from the foreclosure review to the activities under the amended consent orders. In response to the third recommendation, OCC noted that it recognized the importance of providing additional information to the public about the processes, status, and results of the continuing reviews and activities under the amended consent orders. As such, OCC said it plans to issue at least two public reports, as we noted in the draft report. The Federal Reserve stated that it plans to continue to coordinate with OCC to provide consistent guidance during the continuing reviews. As we discussed in the report, developing and using objective measures to monitor and assess consistency among the continuing reviews is important. With respect to the second recommendation, the Federal Reserve stated that it has expanded its planning and monitoring efforts during the course of the foreclosure review and plans to continue to devote resources to planning and monitoring as it implements the amended consent orders. In responding to the third recommendation, the Federal Reserve outlined a number of steps that it and OCC are taking to communicate about the continuing foreclosure reviews and activities under the amended consent orders. These steps, such as developing a letter to explain why borrowers are receiving a payment and updating borrowers covered under the continuing reviews who submitted a request-for-review, will help provide information to affected borrowers. Moreover, the Federal Reserve and OCC said that they have committed to providing public reports that detail the implementation of the agreements. They also said that they anticipate these reports will include information about the findings of completed reviews, the number of requests for review, and the status of other activities under the consent orders. As agreed with your offices, unless you publicly announce the contents of this report earlier, we plan no further distribution until 30 days from the report date. At that time, we will send copies to interested congressional committees, the Board of Governors of the Federal Reserve System, Office of the Comptroller of the Currency, and other interested parties. In addition, the report will be available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-8678 or evansl@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. Key contributors to this report are listed in appendix IV. The objectives of this report were to assess: (1) challenges to the achievement of the goals of the foreclosure review, (2) the extent of transparency in the foreclosure review process, and (3) lessons that could be useful for activities under the amended consent orders and continuing reviews. The scope of our work was limited to the foreclosure review at the 14 servicers that are subject to the April 2011 consent orders. We were in the process of reviewing various aspects of the foreclosure review when the Office of the Comptroller of the Currency (OCC) and the Board of Governors of the Federal Reserve System (Federal Reserve) announced agreements with 11 of the 14 servicers to discontinue the foreclosure review and replace it with a broad payment process. To assess challenges to achieving the goals of the foreclosure review process, we identified three areas of the review process to use as a basis for our analysis. Specifically, we analyzed consultants’ application of guidance to conduct the reviews and recommend remediation; consultants’ development of test questions related to loan modifications, state foreclosure laws, and fee reasonableness; and consultants’ sampling methodologies for selecting files to review. Based on our prior work on the foreclosure review, these were areas we identified as particularly challenging for ensuring consistent reviews. To obtain information on consultants’ development of test questions and application of regulatory guidance, we reviewed the guidance and conducted site visits and in-person interviews with five consultant engagement teams. To identify third-party consultants to interview, we selected consultants that were engaged to conduct reviews for servicers that are overseen by each regulator. In addition, we selected consultants engaged by servicers with a range in sizes of eligible population for the review, including some of the largest servicers. During the site visits we also observed demonstrations of the file review process and systems. In addition, we obtained responses to a standardized questionnaire from all of the consultants. We compared consultants’ processes for compiling reference materials, developing test questions, and applying guidance. We reviewed documents regulators used to monitor the file reviews, such as status reports and meeting agendas. We also interviewed regulator staff and examination teams on steps taken to promote and assess consistency in the reviews and the effects of any differences. To obtain information on third-party consultants’ sampling plans, we used a data collection instrument created by our statisticians. We analyzed information contained in engagement letters between the servicers and consultants for the sampling parameters consultants used to select files. We confirmed key observations of our analysis in interviews and site visits with officials responsible for developing the sampling plans at five third- party consultant engagement teams, interviews with regulator staff, and through examination teams that reviewed the plans. In addition, we obtained information on consultants’ plans for additional analytical methods and confirmed other observations of our analysis through a standardized written questionnaire to consultants. We compared the information and these parties’ actions to criteria such as the regulators’ standard practices, stated goals for the foreclosure review, and our internal control standards. In addition, we reviewed our prior work on the Troubled Asset Relief Program (TARP), Home Affordable Modification Program, and reports where we established criteria on lessons-learned procedures, generally accepted project management practices, and effective management practices. We also referred to our sampling standards and several other references on sampling standards from OCC, the American Institute of Certified Public Accountants (AICPA), and the Office of Management and Budget (OMB). We considered these practices applicable to the foreclosure review because they concern the use of data from samples to draw inferences about the populations from which the samples are drawn. To further evaluate and asses the sampling plans, we also compiled a list of common sampling terms based on the work cited above, as well as other analyses, and vetted the definitions for these terms internally with our methodological staff, including staff responsible for sampling procedures related to financial audits and program audits. Any data we obtained on the number of borrowers in the scope of the reviews or the status of the reviews were used for background purposes only and were not used to support our findings and conclusions. As such, we obtained information from regulator staff and the data administrator about how the data were obtained and compiled, but we did not assess the reliability of the data. Statistical sampling can be a powerful tool for drawing inferences about populations when a full census of cases is infeasible. Well designed and executed samples allow researchers to make estimates of population characteristics and to specify uncertainty due to sampling error associated with those estimates. Parameter estimates and associated measures of precision (such as confidence intervals or margins of error) are developed from the sample data using estimation formulas consistent with the sampling plan actually used. For example, if the sample design were complex (such as using differing probabilities of selection for different portions of the population) and if estimation formulas for simple random sampling were used to produce estimates, those estimates and the associated confidence intervals would likely be incorrect. Additionally, estimates projected to a population of interest may be subject to bias when a sample fails to cover the population of interest, such as when the list used for sampling (the sampling frame) excludes relevant units or when data from only a portion of the sample are collected. The design of an appropriate sample is intricately linked with the goals of the sample. These goals can include the importance of establishing the estimate within a narrow confidence interval, the need to conduct statistical testing against a threshold or for differences, the ability to make estimates to subpopulations, and the desire to report specific results. A sample that is designed to test against a specified tolerance level may use different sample sizes and a different approach to statistical testing than a sample that is designed to estimate the level of an attribute in the population. Some uncertainty is implicit in sampling as a tradeoff for such factors as the cost and time required to examine all of the data. However, according to AICPA guidelines, sampling is inappropriate if there is no tolerance for risk of possible erroneous decisions as a result of examining only a sample of the data. For example, figure 2 demonstrates that these two consultants interpreted the “precision” requirement differently. Consultant A interpreted precision as being the threshold for the tolerable error rate, whereas Consultant B interpreted this as requiring a margin of error within plus or minus 3 percentage points of the estimate. Because Consultant A is testing against a threshold using the methodology specified in the OCC handbook on sampling methodologies (based on a Poisson distribution), it uses the upper bound of its one-sided 95 percent confidence interval to draw inferences about whether the population error rate is less than an established threshold value. Consultant B, in contrast, uses a two-sided 95 percent confidence interval because it is developing point estimates. Although both consultants achieve the 95 percent reliability and 3 percent precision guidance from regulators, the consultants’ different approaches to sampling resulted in different sample sizes and different decision rules for when additional sampling or full review of a population would be required, even when the same number, or even proportion, of errors are found in the sample. For example, if 2 errors were found in Consultant A’s sample, one could conclude with 95 percent confidence that the error rate in the population was below 6.3 percent and that the 3 percent precision threshold had not been met. Additional sampling would be required in this case. If 2 errors were found in Consultant B’s sample, one could conclude with 95 percent confidence that the error rate in the population is between 0.1 and 1.9 percent and no additional sampling would be required. If the proportion of errors in Consultant B’s sample was similar to Consultant A’s at approximately 2 percent, or 8 errors, one could conclude with 95 percent confidence that the error rate in the population was between 1 and 4.3 percent, a much narrower estimate than that from Consultant A. Neither of the samples illustrated in figure 2 was designed with the goal of estimating the number of harmed borrowers in the population. However, had samples of this size been drawn with the purpose of estimating the number of harmed borrowers, the width of the two-sided 95 percent confidence intervals around each estimate would vary. Assuming a population size of 100,000 cases and a binomial distribution, and that each loan with an error corresponds to one harmed borrower, samples similar to those in figure 2 would result in different ability to concisely estimate the number of harmed borrowers. If two errors were found in a sample of 100 files, a 95 percent confidence interval for the estimated number of harmed borrowers in the population would run between approximately 240 and 7,040 borrowers. For a sample of 370 cases, a similar proportion of errors (8 errors in a sample of 370) would result in a 95 percent confidence interval around the estimated number of harmed borrowers between approximately 940 and 4,220, a much narrower confidence interval. To effectively use a baseline sample from the full population of loans to monitor whether the foreclosure reviews have identified the majority of harmed borrowers would require setting a realistic expected error rate to calculate sample size and establishing guidelines for an appropriate margin of error around the estimate to ensure that sample estimates were sufficiently precise to meet regulators’ goals. To assess the extent of transparency in the foreclosure review process, we reviewed press releases and documents from regulators related to the foreclosure review. In particular, we reviewed what documents related to the consent orders were available on the regulators’ websites, such as speeches by agency officials, engagement letters, outreach materials, and press releases, and analyzed the content of these documents. We compared this documentation against agency policies on public disclosure of enforcement action information and our previous work on transparency in government programs, such as our work on TARP to identify any similarities and differences. Further, we reviewed reports and summary documentation from the National Mortgage Settlement and interviewed the monitor of the settlement to provide the context of a current example of a large-scale settlement involving similar stakeholders and issues similar to those of the foreclosure review. We also conducted interviews with regulator staff, selected third-party consultants, and consumer groups. To identify lessons learned that might be useful for the activities under the amended consent orders, we compared our findings from the first two objectives with the processes and goals regulators outlined for the activities under the amended consent orders and identified areas with similar challenges. We reviewed press releases on the agreements and amended consent orders from the regulators and interviewed regulator staff about the goals and rationale for the agreements and their plans for designing and implementing the activities under the amended consent orders. We also reviewed our internal control standards, our prior work identifying key practices during agency transformations, our prior work covering TARP and other government programs and agencies, and documents related to the National Mortgage Settlement. In particular, we focused on information in these reports on project design, oversight of progress, and communication with stakeholders. Although the foreclosure review and the activities under the amended consent orders are not government programs, we considered the steps identified in this work as applicable because planning, monitoring, and communication are key principles of any effective process. We conducted this performance audit from July 2012 through March 2013 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. In addition to the contact named above, Karen Tremba (Assistant Director), Bethany M. Benitez, John Karikari, Charlene J. Lindsay, Patricia MacWilliams, Marc Molino, Jill Naamane, Anna Maria Ortiz, Robert Rieke, Jennifer Schwartz, Andrew Stavisky, Sonya Vartivarian, James Vitarello, and Monique Williams made key contributions to this report.

Summary: Since April 2011, OCC and the Federal Reserve had been overseeing the foreclosure review, a requirement of consent orders entered into by 14 mortgage servicers. This undertaking involved a review of loan files by thirdparty consultants to identify errors in servicing and foreclosure practices. More than 4 million borrowers were eligible for reviews. In January 2013, the regulators announced agreements with 11 of the servicers that replaced the reviews with a broad payment process to compensate borrowers in a more timely manner. Reviews continue for three remaining servicers. GAO has been reviewing various aspects of the foreclosure review process. This report addresses: (1) challenges to the achievement of the goals of the foreclosure review, (2) transparency of the process, and (3) lessons that could be useful for carrying out activities under the amended consent orders and continuing reviews. GAO analyzed third-party consultants' sampling plans, reviewed regulatory guidance and other documents, and interviewed representatives of third-party consultants and law firms, consumer groups, and regulators. Complexity of the reviews, overly broad guidance, and limited monitoring for consistency impeded the ability of the Office of the Comptroller of the Currency (OCC) and the Board of Governors of the Federal Reserve System (Federal Reserve) to achieve the goals of the foreclosure review--to identify as many harmed borrowers as possible and ensure similar results for similarly situated borrowers. Regulators said that coordinating among foreclosure review participants was challenging, and consultants said that the reviews were complex. In spite of regulators' steps to foster consistency, broad guidance and limited monitoring reduced the potential usefulness of data from consultants and increased risks of inconsistency. For example, GAO found that guidance was revised throughout the process, resulting in delays. Other guidance did not specify key sampling parameters for the file reviews and regulators lacked objective monitoring measures, resulting in difficulty assessing the extent of borrower harm. Good planning and collecting objective data during monitoring provide a basis for making sound conclusions. Without using objective measures to assess sampling or comparing review methods across consultants, regulators' ability to monitor progress toward achievement of foreclosure review goals was hindered. Although regulators released more information than is typically associated with consent orders, limited communication with borrowers and the public adversely impacted transparency and public confidence. To promote transparency, regulators released redacted engagement letters and guidance on remediation. In addition, OCC released two interim progress reports. However, some stakeholders perceived gaps in key information and wanted more detailed information about how the reviews were carried out. Regulators stated they considered publicly releasing additional information, but expressed concerns that releasing detailed information risked disclosure of confidential or proprietary information. Further, borrowers who requested reviews experienced gaps in communication. For example, borrowers who submitted requests when the submission period opened waited nearly a year before receiving an update. The foreclosure review activities to date highlight key lessons related to planning, monitoring, and communication. GAO's prior work shows that assessing and using lessons learned from previous experience can benefit the planning of future activities. The foreclosure review produced lessons in advanced planning and establishing mechanisms to monitor progress toward goals. Without assessing and applying relevant lessons learned, regulators might not address challenges in the continuing reviews or similar challenges in activities under the amended consent orders. In particular, regulators announced the agreements that led to the amended consent orders without a clear communication strategy. Although the regulators plan to release reports on the results of the amended consent orders and the continuing foreclosure reviews, neither regulator had made decisions about what information to provide to borrowers. GAO's internal control standards and best practices indicate that an effective communication strategy and timely reporting can enhance transparency and public confidence. Absent a clear strategy to guide regular communications with individual borrowers and the general public, regulators face risks to transparency and public confidence similar to those experienced in the foreclosure review. OCC and the Federal Reserve should improve oversight of sampling and consistency in the continuing reviews, apply lessons in planning and monitoring from the foreclosure review, as appropriate, to the activities of the continuing reviews and amended consent orders, and implement a communication strategy to keep stakeholders informed. In their comment letters, the regulators agreed to take steps to implement the recommendations.