Under legislation that established Advocacy in its current form in 1976, Advocacy’ duties are to serve as a focal point for small businesses’ concerns about the federal government’s policies and activities; advise small businesses on how to interact with the federal develop proposals for federal agencies on behalf of small businesses; represent the views and interests of small businesses before federal agencies; enlist the cooperation and assistance of public and private agencies, businesses, and other organizations in disseminating information about the federal government’s programs and services that benefit small businesses. Since its establishment, a series of laws and executive orders has increased Advocacy’s roles and responsibilities. First, in 1980 the White House Conference on Small Business made recommendations that led directly to the passage of the RFA, which requires government agencies to consider the effects of their regulatory actions on small entities and, where possible, mitigate them. Under the RFA, agencies provide a small business impact analysis, known as an initial regulatory flexibility analysis, with every proposed rule published for notice and comment and a final regulatory flexibility analysis with every final rule. The Chief Counsel for Advocacy was charged with monitoring federal agencies’ compliance with the act and with submitting an annual report to Congress. Second, in 1996 the Small Business Regulatory Enforcement Fairness Act (SBREFA) provided for the judicial review of agency compliance with key sections of the RFA. It also established a requirement that EPA and OSHA convene panels whenever these agencies are developing a rule for which an initial regulatory flexibility analysis would be required (SBREFA panels). These panels consist of the agency, OIRA, and Advocacy. The 2010 Dodd-Frank Act added the newly created CFPB to the agencies required to convene SBREFA panels. The SBREFA panels meet with representatives of the affected small businesses to review the agencies’ draft proposed rules, identify alternative approaches to the rules, and provide insight on the anticipated impact of the rules on small entities. The panels issue a report, including any recommendations for minimizing the economic impact of the rule on small entities. Third, Advocacy’s responsibilities were further expanded by Executive Order 13272, which was issued in 2002. The order required each agency to establish procedures and policies to promote compliance with the RFA and to publish a response in the Federal Register to any written comment received from Advocacy on rules published. This requirement was codified by the Small Business Jobs Act of 2010. Executive Order 13272 also directs Advocacy to provide training to federal agencies on how to comply with the RFA. Until 2010, Advocacy’s budget was part of SBA’s. As of fiscal year 2010, however, Advocacy was given statutory line-item funding in a Treasury account separate from other SBA funding, with Congress setting the amount available for Advocacy’s direct costs. In fiscal year 2014, Advocacy’s enacted budget was $8.75 million. Its fiscal year 2015 budget request was $8.46 million. Of that amount, $7.75 million (92 percent) is to be used to fund compensation and benefits for Advocacy’s professional staff, with the balance of Advocacy’s budget split almost equally between external research and all other direct expenses. Advocacy is organized in five offices, as shown in figure 1. Advocacy’s Office of Economic Research produces both internal and external research, which is publicly disseminated, on a variety of small- business issues. More specialized research—requiring proprietary data or econometric analysis—typically is conducted by contractors (external). SBA handles the contracting process for Advocacy. The contracts generally last for 1 year. Advocacy economists act as the official contracting officers’ representatives (contracting officer) overseeing and coordinating the work of the contractors. The contracting officers maintain a contract file for each external research product. Each year the Office of Economic Research solicits research topics from Advocacy staff and small business stakeholders, such as associations comprised of small businesses. In addition, Congress requests studies, either formally (by putting the requirement into a law), or informally (through discussions with Advocacy staff). A final list of potential research is presented to the Chief Counsel before the beginning of the fiscal year, and the counsel chooses the topics for the year. In fiscal year 2013, Advocacy produced 22 research products on topics that included access to capital, small business exporters, entrepreneurship, and minority- and women-owned businesses. Other Advocacy research addresses the concerns highlighted in Advocacy’s authorizing statute, such as examining the role of small business in the American economy, assessing the effectiveness of existing federal subsidies and assistance programs for small businesses, and evaluating efforts to assist small veteran-owned small business concerns. Advocacy follows a peer review process that was revised and formalized in 2013 and applies to all internal and external research products. Advocacy staff, including the Director of the Office of Economic Research, conduct peer reviews for internal products. For external research products, the peer review is initiated when Advocacy receives the first draft of the product, typically 6 to 8 months after the contract has been awarded, and generally includes external reviewers. In addition to the peer review process, all of Advocacy’s research products are required to pass Advocacy’s internal clearance process, which involves review by the Director of Economic Research, editors in Advocacy’s Office of Information, and individuals in Advocacy’s Office of Chief Counsel (Senior Advisor, Deputy Chief Counsel, and Chief Counsel). Advocacy’s Office of Interagency Affairs oversees the office’s regulatory activities, which aim to convey the views of small businesses on the impact of federal regulations and related costs. These activities generally fall into three categories—developing and issuing comment letters, convening information roundtables, and providing RFA training. Attorneys in this office (“regulatory attorneys”) are expected to become experts in the policy areas they oversee and to establish and maintain broad and effective networks of small business experts (e.g., trade associations) in their policy area. The regulatory attorneys are encouraged to go to trade association and other industry meetings in order to maintain and expand those networks. In addition to maintaining working relationships with industry members and experts, the staff are to establish and maintain relationships with the regulatory staff within each agency who write the rules. One of the primary ways Advocacy provides input to agencies that are issuing rules and regulations of concern to small businesses is through public comment letters. Our review of comment letters from fiscal years 2009 through 2013 found that they covered a wide range of rulemakings on issues such as food labeling, designations for critical habitat, and emission standards. Advocacy made a number of recommendations in its comment letters that included creating an exemption for small businesses or strengthening economic analyses required by the RFA. Advocacy also issued “nonrule” letters that involved the agencies’ other activities, such as agencies’ scientific research. These letters constitute a small proportion of Advocacy’s comment letters. Table 1 below shows the number of comment letters on rulemakings by fiscal year. Regulatory attorneys also convene information-gathering roundtables to discuss the regulatory concerns of small businesses. Roundtables are convened on a regular basis in two policy areas—environment and labor safety—while events covering other areas are convened on an ad hoc basis, depending on which regulatory or rulemaking issues might be forthcoming. According to the regulatory attorneys, the most typical reason for convening a roundtable was an upcoming rule or legislation that would impact small businesses. The regulatory attorneys use the information gathered from the roundtables, together with other information, to inform Advocacy’s positions on the issues involved, and to give Advocacy direction on proposed rules’ economic impacts and possible regulatory alternatives. The attorneys also told us roundtable discussions help them set priorities, and broaden their knowledge base. The roundtables sometimes resulted in a comment letter, although not always. Table 2 shows the number of roundtables by policy area for fiscal years 2009 through 2013. As discussed previously, Executive Order 13272 requires Advocacy to provide training to the agencies on how to comply with the RFA. According to data provided by Advocacy officials, in 2013, Advocacy staff provided training on the RFA to 159 officials at nine agencies and to 22 congressional staff. In addition to the formal training sessions, regulatory attorneys told us they were encouraged to interact regularly with the relevant rulemaking officials at the agencies as rules were developed in order to communicate the concerns of the small business advocates. Producing research products, both internally and externally, on issues of importance to small businesses is one of Advocacy’s primary responsibilities. However, we found that Advocacy’s quality review process lacked some key controls to substantiate the quality of the research and did not take steps to ensure that staff were adhering to existing controls. According to Advocacy officials, peer review is the main quality control over the research it disseminates. Advocacy’s current Chief Counsel recently directed the office to strengthen its peer review process with the intent to make it more rigorous and consistent. As a result, during the course of our review, Advocacy finalized a written peer review process. We found that the written guidance discussed the various levels of review for internal and external products as well as a process for initiating peer review. However, it did not specify how the economists who managed the research products were to identify peer reviewers. Instead, Advocacy officials told us that they relied on their own expertise and professional contacts to identify appropriate peer reviewers and provide recommendations through the Director of Research and to the Chief Counsel for Advocacy. The officials told us that within a specific subject matter there is often a small group of available peer reviewers, in part because their expertise is specialized and there are not many other alternatives. OMB’s peer review guidance calls on agencies to select peer reviewers with the appropriate knowledge and expertise and to take into account their independence and lack of conflicts of interest. Advocacy managers told us that, in practice, the economists recommend peer reviewers based on knowledge and experience in both subject matter and databases—as discussed in the OMB guidelines. However, they did not provide specific written guidance to the economists on how to identify peer reviewers. Federal internal control standards state that internal control activities help ensure that management’s directives are carried out and in implementing those standards management is responsible for developing the detailed policies, procedures and practices for their agency’s operations. With additional guidance, Advocacy would be in a better position to help ensure that the economists fully understand how best to identify qualified peer reviewers and carry out the Chief Counsel’s directive to improve the peer review process. Advocacy does not have consistent documentation showing whether a peer review occurred for all of its research products. Our review of 20 recent research products—10 internal and 10 external—revealed that 16 did not have documented peer reviews in the research files. According to interviews with Advocacy economists who managed the research, all of the 20 products underwent some form of peer review. The economists said that the type of review was commensurate with the methodological complexity of each product, among other factors. However, the Advocacy officials were unable to produce any documentation that peer reviews occurred for these 16 products. According to Advocacy’s peer review guidance, the economists who manage the research should document all correspondence pertaining to the peer review and maintain this documentation in the research file. In addition, federal internal control standards require all transactions and significant events be documented and that the documentation be readily available. Advocacy officials do not have procedures to review the external research files to ensure that the peer reviews were documented. Furthermore, they noted that for some of the less in-depth internal research products— typically 2 to 5 pages—such documentation would be administratively burdensome. However, we note that the documentation could be likewise concise, such as a checklist, or a form that reviewers sign, similar to the one currently used by Advocacy for its internal clearance process. Otherwise, absent written documentation, Advocacy managers are limited in their ability to conduct oversight and ensure that this key quality control activity is happening. For example, for one of the internal research products we selected for review, the study author no longer worked at Advocacy, and therefore no one could tell us with any certainty whether the required peer review had occurred, or who had participated in it. Without adequate documentation—a key internal control—of its peer reviews, Advocacy does not have an institutional record of its activities and cannot demonstrate that it is following its own peer review process. In addition, Advocacy has not consistently documented how peer reviewers’ comments were addressed by the authors of its external research products. Of the 10 external research files we reviewed, 4 had documentation that a peer review occurred, and 1 file included evidence that the peer reviewer comments were incorporated into the final report. Advocacy officials told us that the economists who managed external research consolidated the peer reviewers’ comments and forwarded to the author those that needed to be addressed, including methodological and data issues and other comments, but not those that might change the scope of the contracted research. They also told us that they did not typically maintain documentation showing which comments had been addressed and why, but included the final report in the research file. However, Advocacy’s peer review process states that the economist managing the research will analyze and incorporate, as needed, peer reviewers’ suggestions and maintain all related documentation in the research file. However, since the economists are not keeping records and documenting that comments have been considered and addressed, management does not have an institutional record to provide reasonable assurance that its quality control process is being followed. While Advocacy has quality review policies for its peer review process, it does not have policies and procedures that reflect the federal information quality guidelines on retaining data for influential studies or taking other steps to substantiate the quality of information in such studies when they have not retained the data. Advocacy officials told us that they did not retain the original data or underlying computer codes for three external studies on the costs of regulation as required by the information quality guidelines. We focused on external studies on estimating the costs of regulation because it is a key research area for Advocacy, according to its originating statute and the mission statement of its Office of Economic Research. The OMB Information Quality Guidelines require that all agencies producing and disseminating “influential statistical information” help ensure a high degree of transparency about data and methods to facilitate the reproducibility of such information by qualified third parties. The SBA guidelines implement this standard for transparency by requiring that the underlying data be stored and made available for public review for as long as the agency-disseminated information based on the data are valid. The guidelines also state that all formulas, calculations, matrixes, and assumptions used in processing the data should be available. Because Advocacy classified two of the regulatory cost studies as “influential” according to the OMB guidelines, those data should have been maintained. Advocacy officials said that they did not maintain the data or models for influential external research because there might be a cost associated with obtaining such data, which would raise the costs of the studies, possibly making them prohibitively expensive. However, in the case of two of the studies, the original data were from publicly available sources and involved a relatively small dataset, suggesting the cost would not have been prohibitive. The OMB guidelines state that sufficient transparency—achieved in part by storing the relevant data—results in analyses that can be substantially reproduced. Not retaining the underlying information for these influential research papers makes it much more difficult to assess the quality of that work, including its objectivity—a key goal of the information quality guidelines. We also found that Advocacy staff had not taken additional steps, in the absence of retaining the underlying data, to substantiate the quality of the regulatory cost estimates in two of the studies that it sponsored and disseminated. The OMB guidelines state that when data and methods are not retained and made available to the public because of other compelling interests such as privacy, trade secrets, intellectual property, and other confidentiality protections, the agency shall apply rigorous checks to the analytical results and document what checks were undertaken. Because Advocacy had not retained data on the two cost estimation studies that had been criticized, we interviewed senior Advocacy officials, including the Director of its Office of Economic Research, about the information and methodologies used in the studies. We asked them a set of questions related to criticisms of the methodologies, data, and models used in the studies that were identified in our evaluation and the work of other researchers. Advocacy staff declined to answer many of our questions and instead directed us to the authors, stating that they, not Advocacy economists, were the experts on the issues covered in the studies. However, the authors would not speak with us, stating that they were no longer contractually obligated to respond to our requests for information. During our discussions with Advocacy officials, they stated that the purpose of the studies was not to estimate the overall costs of regulations, but rather to estimate the disproportionate share borne by small businesses. They also noted that, as with all contract research, the external research reports contain a disclaimer indicating that the views presented did not necessarily represent those of Advocacy. In addition, Advocacy added language to the 2010 report’s cover page about the uncertainty of the authors’ estimates of the costs of regulation. Advocacy officials noted that the majority of the research it conducts is not classified “influential” according to the OMB and SBA guidelines, and that they have no plans to engage in such work in the near future. However, given Advocacy’s mission, it may do influential research in the future as it has in the past even if on a limited basis. We acknowledge that these reports may not necessarily be representative of all Advocacy’s research efforts, but not substantiating the quality of the information in even one study could call into question the credibility of Advocacy’s research program. Thus, establishing policies and procedures that reflect the federal information quality guidelines on retaining data for influential studies or when such data are not retained because of certain compelling interests, and requiring additional steps to substantiate the quality of the information, would put Advocacy in a better position to provide reasonable assurance about the quality of its research program. OMB Information Quality Guidelines require that agencies develop policies to ensure that managers be able to substantiate the quality of information they disseminate. The guidelines also discuss narrow circumstances under which an agency does not have to substantiate the quality of the information that resulted from a research project of one of its contractors or grantees. In those circumstances, the researcher is to make clear with an appropriate disclaimer that the views expressed in the research are his or her own and do not necessarily reflect those of the agency. However, OMB cautions that “if an agency, as an institution, disseminates information prepared by an outside party in a manner that reasonably suggests that the agency agrees with the information, this appearance of having the information represent agency views makes agency dissemination of the information subject to these guidelines.” As we noted previously, Advocacy placed a disclaimer on the two studies, but its actions indicate its agreement with the information in the studies. First, Advocacy made the two studies available on its website, where they continue to be available. Second, Advocacy’s disclaimer that appeared on the 2010 study noted that it contained information and analysis that was reviewed and edited by Advocacy. Advocacy’s description of its review role suggests that the agency contributed to the content, if not the conclusions of the study. Finally, several Advocacy comment letters have cited the 2005 study’s regulatory cost estimates in support of their arguments. Because Advocacy’s actions raise, at the least, an appearance of agreement with the information contained in the studies, we believe that the office was required to substantiate the quality of the estimates of the economic costs contained in the studies. Advocacy has practices and procedures (“policies”) for its regulatory activities—comment letters and roundtables—however, documentation of these key regulatory activities is inconsistent. Furthermore, while we determined that transparency and other requirements in FACA do not apply to Advocacy’s roundtables, Advocacy is not following its internal policies meant to ensure its roundtables are as open to the public as they could be. As a result Advocacy cannot demonstrate that it is always fully meeting its mission to foster two-way communication between small businesses and federal policymakers. Our review of Advocacy’s key regulatory activities—developing and issuing comment letters and convening information roundtables—found that Advocacy staff are inconsistently maintaining documentation of key decisions and events. In April 2014, Advocacy provided us with a “practices and procedures guide” for the Office of Interagency Affairs, dated March 2014. The guide covers staff activities related to initiating comment letters and convening roundtables, among other things. Advocacy management told us that they update the guidance periodically, including amending it during the course of our review to make it clear that the decision to issue a public letter or hold a roundtable rests with the Chief Counsel for Advocacy. However, our review found that the updated guide continued to lack policies for documenting key decisions and activities. For example, the guide stated that when regulatory attorneys decide to intervene in the rulemaking process, they must follow up, as appropriate, with the interested associations to ensure that Advocacy has sufficient information and data to make its case for intervening. But there is no policy to document these interactions. The following are other instances where we found a lack of policies to document key activities. Small Business Input into Comment Letters. Advocacy does not have any policies requiring that the regulatory attorneys retain documentation showing which entities provided input into comment letters, and we found that the attorneys do not consistently do so. Our analysis of Advocacy’s comment letters found that about 57 percent of the letters referenced small business input or concerns. But when we interviewed seven regulatory attorneys about how they developed comment letters generally, some told us that they maintained a record of the entities providing input for specific letters, but others said that they did not. Further, when we asked Advocacy staff for documentation on the sources of the small business input referred to in a nonrepresentative sample of 11 comment letters, they were unable to provide specific emails or notes of conversations. Reasons for Convening Roundtables. Advocacy’s practices and procedures guide states that the Chief Counsel must approve the proposed agenda, speakers, and discussion topics for all proposed roundtables before participants are invited. But the guide contains no policy that this step be documented. We interviewed seven regulatory attorneys responsible for the roundtables and their management and were told that the attorneys set agendas and selected speakers based on their own assessment of the issues and, in some cases, with suggestions from small businesses or interested industry parties. Advocacy officials emphasized that they only held roundtables when there is sufficient interest or need on the part of small businesses. Roundtable Discussions and Participants. Information gathered from the roundtables is used to inform Advocacy’s positions on issues related to small businesses and in comment letters, but Advocacy’s guidance contains no policies to document roundtable discussions. Our content analysis of Advocacy’s comment letters showed 19 percent of them referred to roundtable discussions. However, staff did not routinely take and maintain notes of the discussions, according to the interviews we conducted with the regulatory attorneys. In addition, not all staff take attendance at the events. In most cases, Advocacy staff keep an “RSVP list” of those who have indicated that they will attend. However, some of the staff noted that the RSVP lists may not include all participants, including those on the phone. As a result it is difficult to determine the extent to which small businesses and related entities were represented at these events. Advocacy officials told us that they did not have guidance on maintaining documentation on the sources of input to comment letters or on roundtable discussions because they did not feel that setting such standards was required to fulfill their duties under Advocacy’s statute. Furthermore, they noted that, in the case of roundtables, keeping records in a manner that identified specific speakers would inhibit discussion and limit their ability to gain valuable input. They also cited logistical challenges in taking accurate attendance at larger events. We acknowledge that specific parties might not want to be publicly named, but federal standards for internal control call for agencies to document significant events. Taking steps to balance the need for privacy so individuals can speak freely, with a commitment to maintain a basic level of documentation of these events—that is, documenting which entities provided input into its comment letters and roundtables—could help Advocacy demonstrate that it is meeting its mission to represent the interests of small businesses. Key documents for Advocacy’s roundtables—agendas and presentation materials—are not made available to the public at-large after the fact. The regulatory attorneys cultivate email lists of relevant stakeholders who are invited to roundtable events, and these lists are continually updated. It is Advocacy’s policy to add any interested parties to the invitation list, if asked, and several of the attorneys we interviewed said they did so. They also told us they made agendas and presentations available to any interested parties who requested them after the roundtable. However, the agendas and presentations are not posted to the website or made publically available in any systematic way and if small businesses and other interested parties do not know about the roundtables, they cannot request information from the events. Advocacy’s policies and procedures state that agendas and presentations should be posted on Advocacy’s website. Advocacy officials stated they had been unable to post roundtable materials to the website, which SBA maintains, because of difficulties in meeting certain readability and accessibility requirements in the Americans with Disabilities Act. However, a variety of other Advocacy reports and publications are posted on its website that also must meet these requirements, and Advocacy officials did not explain why information on the roundtables could not be likewise included. Making the roundtable materials available on its website would strengthen Advocacy’s efforts to inform small businesses and other interested parties about its efforts to represent their interests to federal decision makers. In addition to evaluating whether Advocacy publicizes and conducts its roundtables in accordance with its internal policies and procedures, we also analyzed whether Advocacy’s roundtable groups constitute “advisory committees” subject to the public notice and other transparency requirements of FACA. We found that the roundtables are not advisory committees, and thus Advocacy is not required by law to follow these rules. We first determined that Advocacy is an “agency” covered by FACA. It is an “authority of the Government of the United States, whether or not it is within or subject to review by another agency.” In addition, Advocacy possesses the type of “substantial independent authority” required by the courts. While located within SBA, Advocacy is independent of SBA, and it has distinct statutory authorities and responsibilities, a separate statutory charter, and an appropriation account that is separate from the rest of SBA. Furthermore, its duties and authorities under statute and Executive Order are substantial and its recommendations made in furtherance of small entities’ concerns must, by law, be given considerable weight by other agencies. While we found that Advocacy is a FACA “agency,” we concluded that its roundtable groups are not “advisory committees” as defined by the statute and interpreted by the courts and implementing regulations. A covered advisory committee is a panel, task force, or similar group created or used by an agency for the purpose of providing advice or recommendations on particular matters. Participants’ input must be sought as a group, not as a collection of individuals. The formality and fixed structure of the group also is an important factor in determining coverage under FACA. Advocacy does not seek roundtable participants’ input as a group, however; rather, attendees provide their individual perspectives on the agency rule or policy under discussion, do not significantly interact with one another, and no attempt is made to reach a consensus. Advocacy’s roundtables also do not have the requisite organized or fixed structure; rather, as noted, Advocacy’s policy is to extend roundtable invitations to anyone who expresses a desire to attend. Finally, according to Advocacy, it is the agency itself, not the roundtable participants, that develops policy advice and recommendations, albeit based in part on the data and information provided by roundtable participants (as well as obtained elsewhere). Our detailed legal analysis of the applicability of FACA to Advocacy’s roundtables is contained in appendix III. Workforce planning presents some challenges for Advocacy, in part because the office is small and has a large number of positions, which, according to Advocacy officials, typically are replaced with each new administration. While the office currently does some workforce planning, its efforts do not include the long-term strategic plans that would help ensure that it maintains the expertise and skills needed to fulfill its mission. While Advocacy now has responsibility for developing its own budget, goals, and performance measures, our review of its strategic plan and goals did not find goals or objectives that discussed workforce issues such as staff development and succession planning. Advocacy officials told us that they used their organizational chart as the basis for their workforce planning efforts and discussed workforce issues with staff and managers at meetings as needed (see fig. 2). Advocacy officials said that because Advocacy was a small office with only 46 staff, the organizational chart and meetings to discuss staffing issues met their workforce planning needs. Advocacy officials told us that to develop staff they provided training to regulatory attorney and economist staff, had established a mentoring program for newer staff, and were implementing a knowledge-sharing database to help staff develop expertise in specialized policy areas. When we asked Advocacy officials about succession planning and their plans for addressing the departure of senior or experienced staff, the officials told us that most of the turnover in staff occurred during changes in administration. For example, according to Advocacy officials, the change of administration in 2009 resulted in the turnover of at least 15 of 20 senior officials. Among these were the 10 regional advocates, who are located in each of SBA’s regional offices across the country, and make up the Office of Regional Affairs. Turnover tends to be low among the economists and regulatory attorneys, some of whom have been with the office for years. Advocacy officials said that because rulemaking often took so long, they generally had enough time to hire or realign any staff as needed. According to federal internal control standards, effective management of a workforce is essential to achieving program results. Our body of work on workforce planning has demonstrated the importance of such planning and the need to develop long-term strategies—such as training and succession planning—for acquiring, developing, and retaining staff to achieve programmatic goals. OPM’s Human Capital Assessment and Accountability Framework states that agencies and offices with workforce planning are better able to manage their staff by, for example, ensuring that systematic processes are in place for identifying and addressing any gaps between current and future workforce needs. Further, OPM recommends that succession planning is needed to ensure continuity in leadership positions. In addition, workforce planning can help management determine the type of training and other strategies that are needed to address factors such as projected retirements and succession planning. In past work on succession planning, we have found that in addition to focusing on replacing individuals, succession planning strategies can also strengthen current and future organizational capacity. Given the length of time some attorneys and economists have been with Advocacy, the loss of their expertise through retirement, among other things, could leave significant gaps in needed skills and knowledge, according to Advocacy officials. Yet Advocacy has not incorporated succession planning into its workforce planning efforts, such as its training and mentoring initiatives. Further, Advocacy officials told us that they discussed future staffing needs and various options for addressing them, including training, but these efforts are not documented in a manner that would ensure consistent implementation. However, ensuring that activities such as staff training are consistently implemented is particularly important when senior management can change significantly every 4 or 8 years. Without incorporating long-term succession planning into its workforce planning efforts, Advocacy is not in the best position to ensure that it has qualified staff to fill leadership and other key positions and a skilled workforce able to meet the demands of its mission on behalf of small businesses. Effective internal controls are critical for the Office of Advocacy if it is to achieve program outcomes and minimize operational problems. Recognizing this, Advocacy recently has taken action to improve some of its guidance and controls. However, Advocacy’s research, regulatory, and workforce planning functions could be improved by strengthening its internal controls as follows: Research activities. In its research operations, Advocacy has taken some initial steps toward establishing stronger control policies by, for example, formalizing its peer review process for internal and external research products. However, the guidance provided did not include information on how to select appropriate peer reviewers—the experts whom Advocacy relies on to assess the quality of the research that it disseminates. The guidance also lacks policies for documenting that a peer review occurred and how reviewer comments were addressed for its external research products. Such guidance would help ensure that Advocacy research staff fully understand how best to identify the most qualified peer reviewers, and how to document and incorporate reviewer comments. In addition, Advocacy did not follow federal guidelines for information quality for influential studies that set out requirements for retaining data and taking additional steps to substantiate the quality of information it disseminates when it has certain compelling reasons to not retain the data. Improving policies and procedures for its research activities would help support Advocacy’s mission to provide quality research on small business issues that decision makers and the public can rely on. Regulatory activities. Advocacy’s lack of documentation and transparency of the regulatory activities we reviewed made it difficult to validate Advocacy’s efforts to represent small businesses. Specifically, weaknesses exist in Advocacy’s documentation of both the sources of the small business input in comment letters and the views of small businesses discussed and conveyed at its roundtables. As a result, the extent to which the small businesses’ views on regulations were being obtained and communicated is unknown. Improving its guidance to staff on its regulatory activities and emphasizing the importance of documentation would enable Advocacy to more effectively demonstrate to decision makers that it was obtaining and communicating the interests of small businesses. In addition, Advocacy is not following its internal policy to post materials from roundtables on its website. As a result, it is missing an opportunity to reach out more broadly to small businesses and other interested parties and increase the transparency of its activities. Workforce planning activities. While Advocacy’s workforce planning efforts help in managing its current staff, these efforts do not include any strategies to plan for succession, even though several staff have been with the office for many years and will eventually need to be replaced. Without this important element of workforce planning, Advocacy could be in a vulnerable position when critical staff leave the agency or staff face new demands. Although Advocacy is a relatively small office, having a skilled workforce is critical to meeting its mission. Succession planning would help ensure that Advocacy was better prepared to maintain qualified staff to conduct its research and regulatory activities in support of small businesses. To improve Advocacy’s system of internal control, and help to provide reasonable assurance that the office is meeting its mission, we recommend that the Chief Counsel of Advocacy take the following five actions: Strengthen the accountability of its research activities by taking the following two actions: Enhance its peer review policies and procedures by including written guidance on selecting peer reviewers and documenting all steps of the peer review process—whether a peer review occurred and how and to what extent peer reviewer comments were addressed. Develop policies and procedures that reflect the federal information quality guidelines on retaining data for influential studies, and when not retaining data, taking additional steps to substantiate the quality of information disseminated. Strengthen the accountability of its regulatory activities by developing policies and procedures to ensure that key elements of that work— such as sources of input for comment letters and roundtable discussions—are consistently documented. Coordinate with SBA officials who oversee website administration to comply with Advocacy’s roundtable policy to make information on the events—agendas, presentation materials—publicly available on its website so that its regulatory activities are more transparent to the public. Improve its workforce planning efforts to be better prepared to meet its future workforce needs by incorporating succession planning. We provided a draft of this report to Office of Advocacy for its review and comment. In its written comments (reproduced in app. II), Advocacy agreed with our recommendations. Advocacy stated that for its research activities, its current effort to further formalize procedures for the peer review process will include steps for additional documentation and that it also plans to develop written guidelines for determining which research products are considered influential, which will clarify when Advocacy needs to take additional information quality steps. Advocacy also said that as it develops how it will disseminate information about its regulatory activities, it will incorporate approaches that are responsive to our recommendations. Finally, Advocacy agreed that workforce planning is important for ensuring that the office maintains the skills and resources needed to fulfill its mission and noted that the office is developing a Leadership Succession Plan in response to our recommendation. We are sending this report to the Office of Advocacy and interested congressional committees. The report also will be available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact Cindy Brown Barnes at (202) 512-8678 or brownbarnesc@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made major contributions to this report are listed in appendix V. This report examines Advocacy’s (1) research activities; (2) regulatory activities, including the applicability of the Federal Advisory Committee Act (FACA) to Advocacy’s roundtables; and (3) workforce planning efforts. The scope of our work covered key activities conducted from fiscal year 2008 to fiscal year 2013, although in some cases, we broadened that scope to include fiscal years 2005 through 2008, as noted below. To understand Advocacy’s mission, operations, and participation in the rulemaking process, we reviewed relevant laws and regulations, Advocacy documents, including its annual reports to Congress on the Regulatory Flexibility Act (RFA), budget and strategic planning documents, and other publications. To assess Advocacy’s research activities, we determined: how Advocacy staff chose its research topics; how they conducted the research; and what controls they had in place to ensure quality products. We analyzed Advocacy’s research products produced in the 5 most recent fiscal years—2008 through 2012—and interviewed relevant Advocacy officials on the processes by which they assess the quality of its research products. To evaluate Advocacy’s research activities, we analyzed a nonprobability sample of research, which included 10 research products authored by Advocacy staff (internal), and 10 products produced by its contractors (external). To select a sample of internal research products that best represented the variety of internal research, we reviewed the 58 research products issued from fiscal years 2008 through 2012, and categorized each as either “routine” (meaning published either quarterly or semi-annually, and relying on the same underlying data set), or as “nonroutine.” Within the routine category, we selected the mostly recently published product from each of the following routine publications: (1) Small Business Profiles of the States and Territories, (2) Small Business Quarterly Bulletin, (3) Quarterly Lending Bulletin, (4) Small Business Lending, and (5) The Small Business Economy. For the nonroutine products, we created five categories, such as “analysis” and “fact sheets.” We selected, from each of those five categories, the most recently published report. Then, for each of the 10 selected internal research products, we used a data collection instrument and interviewed the Advocacy staff who wrote the products in order to see to what extent the products adhered to Information Quality Guidelines issued by the Office of Management and Budget (OMB) and the Small Business Administration (SBA) and Advocacy’s internal quality review process (“peer review”). To assess the quality of the peer review used for Advocacy’s external research, we reviewed the contract files for the 10 most recently published external research products, as of year-end 2013, and compared how the work was produced, including Advocacy’s peer review process, against the OMB Peer Review Guidance and OMB and SBA information quality guidelines, and applicable federal internal control standards. The results from our reviews cannot be projected to all Advocacy studies, but they provide an indication of how Advocacy staff conducted or oversaw the research and what controls were in place to ensure quality products. We also assessed Advocacy’s contract studies that focused on the economic costs of regulation. We focused on external studies in these areas because they are key research areas for Advocacy, according to its originating statute and the mission statement of its Office of Economic Research. For the economic costs studies, we assessed them against OMB and SBA Information Quality Guidelines, and examined Advocacy’s compliance with related data retention policies therein. We also reviewed peer reviewers’ comments and other external reviews on the studies as part of our assessment. Lastly, we interviewed Advocacy officials to obtain information on steps they took to substantiate the quality of information for influential studies when data are not retained. To evaluate Advocacy’s regulatory activities, we assessed: how and why Advocacy decides to issue comment letters and convene roundtables; the policies and practices in place that pertain to those activities; how Advocacy staff solicit input from small businesses and other parties. We interviewed relevant staff, as described below; reviewed relevant policies and procedures, and analyzed comment letters produced and roundtables convened from fiscal years 2009 through 2013. Specifically, we performed a content analysis on the 181 comment letters issued during that period, using NVivo software, to analyzed and categorize the content of the letters and the nature and source of the small business input provided. We also interviewed Advocacy staff who are responsible for rulemaking and related activities, weighting our choices of interviewees to reflect the distribution of comment letters by policy area, and reviewed supporting documentation for comment letters to understand how Advocacy staff develop and issue comment letters, and compared this information to federal internal control standards. Similarly, we reviewed available information on the 142 roundtables convened during the same 5-year period and interviewed the responsible Advocacy staff, weighting our choices of interviewees to reflect the distribution of roundtables by policy area. We reviewed Advocacy’s policies for roundtables to understand their origin, purpose, and documentation requirements, and compared this information to federal internal control standards. In order to understand the perspective of those who attended the roundtables, we attended three of the events. In addition, we selected and interviewed a nonprobability sample of past participants from email lists provided by Advocacy’s regulatory attorneys. Specifically, we talked to the following representatives: five from industry associations that represent small and large businesses; one from a large corporation; and one nonprofit organization whose mission relates to policies or rules under consideration. We made our selections to include representatives from a variety of sources, and while the results from our interviews cannot be projected to all entities that interact with Advocacy, the information we gathered does provide insights into how the selected groups view Advocacy and its work in representing small businesses to federal policymakers. We also reviewed Advocacy’s training to agency staff in how to comply with the RFA and interactions with agency rulemaking officials. In addition, we interviewed officials from the entities that interact with Advocacy in rulemaking—the Environmental Protection Agency (EPA), Occupational Safety and Health Administration (OSHA), Bureau of Consumer Financial Protection, also known as the Consumer Financial Protection Bureau (CFPB), and Office of Management and Budget’s (OMB) Office of Information and Regulatory Affairs (OIRA). To analyze whether FACA (5 U.S.C. App. II) applies to Advocacy’s roundtables, we reviewed relevant statutes, case law, regulations, and guidance. In addition, we reviewed and considered Advocacy’s written views on the issue. See appendix III for more information on the legal analysis we conducted. Finally, we reviewed Advocacy’s workforce efforts. To understand Advocacy’s workforce planning, we reviewed Advocacy’s strategic goals and other planning and budget documents and interviewed senior management to determine what, if any, policies and procedures were in place related to workforce and succession planning. We assessed any such policies and procedures against applicable federal standards for internal control and the Office of Personnel Management’s (OPM) Human Capital Assessment and Accountability Framework (HCAAF). We also reviewed GAO reports on workforce and succession planning to gain insights about key practices and how agencies have used them. We conducted this performance audit from August 2013 to July 2014 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. Advocacy is an “agency” under FACA first because it meets the statutory definition of this term in FACA and the Administrative Procedure Act incorporated by reference: it is an “authority of the Government of the United States, whether or not it is within or subject to review by another agency” and not subject to statutory exemption. In addition, Advocacy has the type of “substantial independent authority” and other indicia of “agency” status that courts interpreting the APA definition have required. However, Advocacy’s roundtable groups are not FACA “advisory committees” because they do not have an organized or fixed structure; there is no attempt to reach consensus about an agency regulation or policy but instead viewpoints are sought from individual participants; and it is only data and information, according to Advocacy, rather than advice or policy recommendations, that participants provide. Cindy Brown Barnes (202-512-8678), brownbarnesc@gao.gov. In addition to the contact named above, Kay Kuhlman and Triana McNeil (Assistant Directors), Kristeen McLain (Analyst-in-Charge), Emily Chalmers, William Chatlos, John McGrail, Lauren Nunnally, Roberto Pinero, Jena Sinkfield, Jack Wang, and Carrie Watkins made key contributions to this report.

Summary: Congress created Advocacy within SBA in 1976 as an independent voice for small businesses. Questions have recently been raised about Advocacy's efforts to represent small businesses in regulatory activities and some of its research on small business issues. In light of these questions, GAO was asked to review Advocacy's activities. This report examines Advocacy's (1) research, (2) regulatory activities, and (3) workforce planning efforts. GAO analyzed Advocacy's research, comment letters, and other regulatory information for fiscal years 2008-2013; assessed Advocacy's policies and procedures against federal standards for internal control and information quality; and interviewed agency officials, and small business and industry representatives. GAO also analyzed the applicability of the Federal Advisory Committee Act to Advocacy roundtables. The Office of Advocacy (Advocacy) within the Small Business Administration (SBA) fulfills its mission by researching small business issues and providing input into federal rulemaking and related regulatory activities. However, GAO identified key areas in Advocacy's system of internal control that could be improved. Research. GAO found that Advocacy did not ensure that its staff monitored the quality of the information the office disseminated, as required. GAO reviewed 20 selected research products and found that in 16 cases a required quality review had not been documented. Advocacy recently established a review policy for its research, but it does not include procedures for selecting the reviewers or documenting that a review occurred and how reviewer comments are addressed. GAO also found that Advocacy staff had not followed federal information quality guidelines to retain data and could not substantiate the quality of information in two cost-estimation reports—a research product it has contracted for every 5 years. Without better controls over its quality review process and efforts to substantiate the information it disseminates, Advocacy cannot ensure the validity of one of its core activities—research in support of small businesses. Regulatory activities. Advocacy recently updated procedures for its regulatory activities, but these could be strengthened. GAO found the extent to which individual staff maintained records varied, in part, because the procedures lacked policies for documentation. For instance, the procedures state that when staff decide to intervene in the rulemaking process, they must follow up as appropriate with the interested groups to ensure that Advocacy has sufficient information and data to support its case. However, there is no policy that these interactions be documented. Federal internal control standards state that documentation and records should be maintained. If key procedures are not being documented, managers do not have an institutional record that agency goals and objectives in this area are being met. GAO also found that the Federal Advisory Committee Act's transparency and other requirements do not apply to Advocacy's meetings with stakeholders to get input on regulations (roundtables). Workforce planning. Advocacy's workforce efforts include training and mentoring for new staff, but do not include succession planning, which is recommended by the Office of Personnel Management. According to federal internal control standards, effective management of a workforce is essential to achieving program results. Officials told GAO that Advocacy was a small office and that additional staff were hired on an as-needed basis. However, some key staff have been with Advocacy for many years and their experience will be difficult to replace. If Advocacy does not incorporate succession planning strategies into its workforce planning efforts, it is at risk of not having the skills and expertise to meet its mission when key staff leave or retire. GAO makes several recommendations to improve the Office of Advocacy's controls over the quality of its research, the documentation of its regulatory activities, and workforce planning. In commenting on a draft of this report, Advocacy agreed with our recommendations and noted some steps it will take to address them.