FMCSA, established as a separate administration under DOT by the Motor Carrier Safety Improvement Act of 1999 is responsible for improving the safety of commercial vehicle operations on the nation’s highways. FMCSA is engaged in several programs and activities to carry out its mission, including developing and enforcing Federal Motor Carrier Safety Regulations, administering Motor Carrier Safety Assistance Program grants to states, regulating interstate household goods movers, and performing education and outreach. FMCSA operates through its headquarters in Washington, D.C.; four regional service centers; and division offices located in all 50 states, the District of Columbia, and Puerto Rico. FMCSA undertakes education or outreach while carrying out many of its functions—for example, in posting regulations pertaining to commercial drivers’ licensing or transporting hazardous materials on its Web site, or in providing on-line access to motor carrier crash statistics. FMCSA is also allocated funding for specific education and outreach programs. In fiscal year 2005, FMCSA used education and outreach funding for activities supporting its Safety Belt program targeted to commercial vehicle drivers, outreach to the public on hiring motor coach services, and outreach to consumers using household goods movers. Congress also allocated fiscal year 2005 funding for FMCSA’s program directed at recently registered new entrant motor carriers and funding to establish an initiative directed to non-entrants—which are carriers in interstate operations that have not registered with DOT. Under the Government Performance and Results Act of 1993 (GPRA), federal programs should be designed with measurable goals that support the agency’s overall strategic goals. Congress enacted GPRA to shift agencies’ focus from simply monitoring activities undertaken to measuring the results of these activities. Under GPRA, agencies develop multiyear strategic plans, which are the starting point for their performance measurement. Each strategic plan is to include a mission statement, a set of outcome-related strategic goals, and a description of how the agency intends to achieve these goals. To measure progress toward the strategic goals, we have previously reported that the agency should also have a plan for collecting data to measure and evaluate program performance. Without measurable goals and evaluation, it is difficult to determine whether the program is accomplishing its intended purpose and whether the resources dedicated to the program efforts should be increased, used in other ways, or applied elsewhere. Research conducted on the effectiveness of media campaigns indicates that the development of an education and outreach program should include identifying how the program is expected to change the target audience’s attitudes and behaviors. There are several theories and models on affecting behavior, and two in particular are the most relevant to the education and outreach programs under this review. This is because the theories are consistent with the structure and goals of the FMCSA programs we reviewed. One behavioral model that researchers have developed—the reasoned action theory—assumes that people will take a desired action if they believe the action is beneficial, and they have the means to undertake it. Another model that experts have developed—the general deterrence theory—indicates that in some circumstances people will take a desired action when they recognize that they may be punished for failing to act. The relative roles of education and enforcement depend upon the program’s target audience, the desired change to be achieved, and other factors. Since media campaigns should identify how the program is expected to change target audiences’ attitudes and behaviors, program managers can use a tool or framework to show how outreach activities are intended to influence attitudes and behaviors, and thus achieve broad program goals. Academic literature on program development and evaluation provides a number of models. One tool we have previously used to review education and outreach programs is called a logic model. A logic model links program inputs and outputs to program outcomes (see fig. 3). Agencies have used such a model to plan, evaluate, and adjust program activities. A logic model can facilitate planning and evaluation by (1) demonstrating accountability through focusing on measurable outcomes; (2) linking activities to results to prevent mismatches between program activities and outcomes; and (3) integrating planning, implementation, evaluation, and reporting. In addition, a program logic model can help describe a program’s components and desired results and explain the strategy—or logic—by which the program is expected to achieve its goals. The logic model can help clarify the links between program components, focusing on outcomes that are measured and under some degree of control. In some cases, it may be difficult to show a direct link—a specific cause and effect—between program activities and outcomes due to the influence of external factors. For example, the Safety Belt program, although targeted toward commercial motor vehicles operators, may have outcomes influenced by the NHTSA’s “Click It or Ticket” program targeted to passenger car drivers. Commercial motor vehicle operators could be affected by this program, or other safety-belt messages, and decide to wear safety belts when driving commercially. Thus, the Safety Belt program’s initiatives may not have been the primary factor in determining a particular driver’s safety-belt use. Finally, once program activities are established and linked to program goals, organizations can use program evaluation to determine whether or not a program is meeting its goals. Program evaluations are conducted periodically to provide an overall assessment of how well a program is achieving its expected results. We have reported on the value of program evaluations to assess the impact of a particular program. This work showed that results of evaluations can support decisions on resource allocation and ways to improve program effectiveness. FMCSA has established several education and outreach programs, with different purposes targeted to different audiences. Total funding for these programs in fiscal year 2005 was $36.3 million, with the largest share—$33.1 million—allotted to the New Entrant program. FMCSA’s education and outreach efforts are intended to address recognized problems by communicating information to motor carriers, commercial drivers, and the public to encourage safer practices and better decision making. FMCSA activities under these programs range from distributing brochures and posting information on Web sites, to site visits to newly registered motor carriers under its New Entrant program. Although FMCSA officials consider these programs to be linked to agency goals at a high level, this linkage to broader agency goals is not explicit in the agency’s planning and budgeting documents. FMCSA officials state that they have used a logic model in their performance budgets to illustrate the link between education and outreach and agency goals at a high level. Although the performance budget implies a link between the education and outreach program and agency goals, it does not describe how the activities are intended to support these goals. As a result, it is difficult for stakeholders to see how education and outreach activities that seek changes in attitudes and behavior will ultimately contribute to agency goals. Also, without a clear link, program managers can not easily determine if program activities are appropriately targeted or if they need to refine their programs to meet agency goals. In contrast, we found, however, that FMCSA uses a logic model in the strategic plan for its Research and Technology group to demonstrate how its programs and activities support agency goals. As FMCSA continues its efforts to develop the links between its education and outreach activities and broader goals, it might consider using such a logic model to make those relationships clear. We reviewed five education and outreach programs that FMCSA has initiated in response to either congressional direction or concerns about motor carrier operations. FMCSA’s education and outreach programs target a variety of audiences, including the motor carrier industry, commercial vehicle drivers, and the public; and they involve a variety of approaches, such as direct contact with carriers, media campaigns, distributing printed materials, and establishing Web sites. FMCSA also works with industry and law enforcement associations, other agencies, and safety-oriented organizations in carrying out some of these education and outreach efforts. Four of these programs are about motor carrier, driver, or passenger safety, while one has a consumer information focus. Table 1 provides a brief overview of the target audience, general purpose of the program, and program activities. The five programs in this review are summarized below, and discussed in greater detail in appendix II. FMCSA intends the New Entrant program to inform newly registered motor carriers (new entrants) about motor carrier safety standards and regulations to help them gain compliance with FMCSA requirements and improve truck safety and thus reduce crash rates. Studies have shown that new carriers have higher safety violation and crash rates than more experienced carriers. FMCSA uses its Education and Technical Assistance Package to communicate safety requirements and other information to new entrants when they register with DOT. This information package is also available at FMCSA’s Web site, along with carrier registration forms. The primary activity of the New Entrant program is safety audits conducted through site visits to the new carriers. Safety audits are conducted by FMCSA, state law enforcement, or contractor personnel, who review the carrier’s compliance with FMCSA requirements, including driver qualifications, driver records of duty status, vehicle maintenance records, and participation in a controlled substance and alcohol use testing program. As of April 2005, FMCSA had conducted 52,000 safety audits since the New Entrant program began in 2003. FMCSA is developing a Non-Entrant program intended to inform start-up motor carriers who have not registered with the agency—called non-entrants—of the requirement to register with FMCSA. FMCSA officials are concerned that carriers that have not registered with FMCSA may represent an increased safety risk. FMCSA expects to target motor carriers who have not registered as interstate carriers with FMCSA, by working through truck sales and leasing firms to inform start-up motor carriers about registration requirements and, ultimately, to ensure that the carriers register as new entrants. FMCSA officials expect to award a contract to develop the Non-Entrant program before the end of 2005. The Commercial Motor Vehicle Safety Belt program is intended to increase safety-belt use by commercial truck drivers. FMCSA began this effort after a study showed that only 48 percent of commercial truck drivers used safety belts, compared with nearly 80 percent safety-belt use by passenger car drivers. Working through the Commercial Motor Vehicle Safety Belt Partnership—an organization of government, industry, law enforcement, and safety associations—FMCSA facilitates and coordinates the development and distribution of outreach materials, including posters, bumper stickers, and brochures promoting safety-belt use. (See fig. 1 for an example of a brochure describing the safety-belt partnership.) Working through the Partnership allows FMCSA to leverage its resources and tap into organizations that have their own communication links to the trucking industry. FMCSA staff distribute materials through trucking industry shows and events, and state law enforcement personnel distribute materials during safety enforcement campaigns. FMCSA has prepared over 200,000 brochures and 50,000 bumper stickers for the 2005 “Be Ready, Be Buckled” campaign. FMCSA also provides safety belt information through its Web site. to help them select and hire motor coach services. The program began following an increase in motor coach accident fatalities. Currently, FMCSA’s primary effort is maintaining a Web site that allows individuals to access advice on selecting a motor coach company and safety information on specific motor coach companies. For the “Moving Kids Safely” campaign in 2002 that was part of the Motor Coach Outreach program, FMCSA created 49,000 brochures; 1,200 posters; and 1,200 “toolkits” to provide advice to school districts on hiring school bus services. FMCSA also reaches out to the industry to inform motor coach companies of regulations and provide safety advice through printed materials and the agency’s motor coach specialists. The Household Goods Outreach program is intended to help consumers make more informed choices in selecting an interstate moving company to avoid unscrupulous movers. This program was begun in response to congressional concern over increasing consumer complaints about fraudulent moving companies. FMCSA distributes printed material through moving companies and other means, such as its Web site and the General Services Administration (GSA) consumer information catalog, to inform consumers of their rights when dealing with moving companies. The material that FMCSA has developed includes a new outreach theme: “Protect Your Memories, Your Money, Your Move” and it includes brochures and presentations that can be delivered by FMCSA staff. For example, as of October 2005, FMCSA officials reported that they had printed a total of 50,000 copies of the two brochures the agency distributes. FMCSA also maintains a hotline to receive complaints from individuals about problems with movers and enters them into a database. Although FMCSA does not get involved in resolving complaints, the agency uses the database to target firms for investigation. Also, we previously reported that FMCSA should make the complaint information available to consumers. According to FMCSA officials the agency has not yet done this because of privacy and other legal issues. The five education and outreach programs represent a small portion of FMCSA’s total funding. Fiscal year 2005 funding for the five programs totaled $36.3 million, out of approximately $443.3 million to fund FMCSA operations and grant programs. The New Entrant program accounted for $33.1 million of the funding for the five programs, while funding for the other four programs combined was $3.2 million—less than 1 percent of the agency’s total fiscal year 2005 funding. The first chart in figure 2 below depicts the New Entrant program, and the combined funding for the other four programs, as their relative share of total FMCSA funding in fiscal year 2005. The second chart depicts the relative size of the four smaller education and outreach programs. Other four programs ($3.2M) Total New Entrant ($33.1M) Motor coach ($150,000) Safety Belt ($500,000) Remaining FMCSA ($407M) Non-Entrant ($1.05M) Household Goods ($1.5M) The funding available for education and outreach supports a diverse range of activities, from the costs of state personnel conducting safety audits, to the costs of printing and distributing brochures and other materials, and the costs of contracts being awarded to develop and evaluate education and outreach activities. The principal activities that FMCSA is undertaking through the five programs are shown in table 2. Congress has expressed concern about how FMCSA’s education and outreach activities are meeting broader goals, such as individual program goals, or DOT’s strategic goals. The conference report accompanying the DOT appropriations bill for fiscal year 2005, for example, asked that FMCSA report by April 2005 to the House and Senate Committees on Appropriations on strategies linking outreach and education program initiatives to each goal. FMCSA sent its report on October 11, 2005. The report indicates that FMCSA believes education and outreach programs support the overall DOT safety strategic objective of “enhancing public health and safety by elimination of transportation deaths and injuries.” FMCSA also indicates that every education activity undertaken “is vital to FMCSA’s goal of reducing fatalities, injuries, and crashes, and attaining a large truck crash rate of no more than 1.65 fatalities per 100 million miles of truck travel by the end of 2008.” Although the report was intended to address congressional concerns about linking education and outreach to program goals, it is unclear specifically how this will be accomplished, since the report does not describe how these programs are intended to influence knowledge, attitudes, and behaviors and thus support FMCSA’s broader goals. Although the report describes in detail the activities FMCSA is undertaking for each of these programs, it does not describe how these activities are intended to change attitudes and behaviors that will ultimately meet FMCSA’s goals. In addition, although FMCSA’s fiscal year 2006 performance budget has some discussion of how activities support FMCSA and DOT safety and productivity goals, except for the Commercial Motor Vehicle Safety Belt program, none of the budget or program documentation we reviewed for FMCSA’s education and outreach programs provides detailed descriptions about how activities link to goals. A greater level of detail in the description of how education and outreach program activities link to goals would help the public and other interested parties to see how program activities that seek changes in attitudes and behavior will ultimately contribute to agency goals. Furthermore, without this link, program managers cannot easily determine if program activities are appropriately targeted or if they need to refine their programs to meet agency goals. FMCSA officials stated that information about education and outreach programs is reported in DOT’s strategic plan and in FMCSA’s performance budget documents. However, DOT’s strategic plan does not specifically address FMCSA’s education and outreach programs and refers readers to the performance budgets for more detailed, program-specific information. FMCSA’s performance budget documents identify strategic objectives, such as saving lives and reducing injuries by preventing bus and truck crashes, that link and contribute to objectives in DOT’s strategic plan, such as reducing transportation-related fatalities. It also identifies underlying performance goals and indicators for each objective. For example, FMCSA’s strategic objective for safety is illustrated in table 3. The performance budget documents address education and outreach primarily under this strategic objective for safety, by stating that “educating carriers about the benefits of operating safely, and in compliance with safety regulations, is advantageous to both the carrier and enforcement community.” The budget also states that the education programs are designed to change “the knowledge, attitudes, and behaviors of commercial motor carriers, commercial motor vehicle drivers, and passenger vehicle drivers, driving in the vicinity of large trucks.” However, the performance budget does not provide specific information for each program on how these expected attitude and behavior changes are linked to broader goals. Of the five programs included in our review, the description of the safety belt initiative provides the clearest information articulating how the program is intended to affect the knowledge, attitudes, and behaviors of its targeted audience, and consequently to improve safety. FMCSA describes how education will highlight the risks of not wearing a safety belt, which in turn is intended to improve drivers’ attitudes toward wearing safety belts, and subsequently meet the program goal of increasing safety-belt usage 10 percent by 2009 and further, meet the strategic objective of reducing fatalities. The performance budget also indicates that education will be combined with traffic enforcement of safety-belt requirements to achieve this outcome. In contrast, the performance budget discusses the New Entrant program in terms of staffing, funding, and the number of safety audits to be done in fiscal year 2006, without defining a goal for what the New Entrant program will accomplish and how that accomplishment would contribute to the strategic objective of reducing fatalities. In addition, the links between the Motor Coach and Household Goods program activities and broader strategic goals are difficult to discern. Specifically, the performance budget document explains that the Motor Coach program seeks to increase consumers’ knowledge about the safety records of carriers, with a potential side benefit of encouraging motor carriers to maintain good safety records. FMCSA does not, however, describe how affecting consumers’ choice of a particular motor coach company would support the overall strategic objective of reducing transportation-related fatalities. FMCSA’s performance budget relates its Household Goods program to DOT’s mobility and FMCSA’s productivity objectives. The mobility objective is “to advance accessible, efficient, and intermodal transportation for the movement of people and goods.” The productivity strategic objective is “to promote efficient and economical motor carrier operations to sustain mobility and economic growth.” The performance budget document asserts links between these objectives and the education and outreach program, but it does not explain how specific activities—such as establishing its Web site on selecting moving companies and creating brochures on consumer rights and responsibilities—support either of these objectives. On the basis of our discussions with FMCSA officials, it appears the agency is attempting to more clearly describe how education and outreach activities link to the agency’s goals. FMCSA acknowledged that there may have been some disconnection between the strategic planning and budgeting processes in the past, but managers have recently received training on defining a program’s desired outcome and then linking inputs—such as funding—and outputs. Officials told us that as a result the agency’s proposed fiscal year 2007 performance budget will better link programs and goals to budget requests. FMCSA’s eventual goal is to link program outputs to outcomes, as well as identifying measurable and verifiable goals to the extent possible. Although the relationships among program activities, changes in knowledge and behaviors, and broader goals can sometimes seem intuitive, laying a program out through a logic model can be useful for program managers and stakeholders to demonstrate and understand these relationships. FMCSA officials state that they use a logic model in its performance budget to link education and outreach to agency objectives. However, the performance budget implies a link between the education and outreach program budget and agency goals without describing how FMCSA intends the program to change behaviors, such as following federal safety regulations, to support agency goals, as would be shown in a logic model. FMCSA has used a logic model in another program area that provides a description of how a program is intended to support agency goals. Specifically, FMCSA’s Research and Technology group used a logic model along with other planning tools to demonstrate how its activities support FMCSA’s goals in its 5-Year Strategic Plan. The Research and Technology logic model includes inputs such as staff, outputs such as research studies, outcomes such as improved understanding of root causes of crashes, and impacts such as fewer commercial motor vehicle crashes. FMCSA described the Research and Technology Logic Model as a “mechanism to help define metrics for performance, quality, and relevance of the program.” As FMCSA endeavors to describe how its education and outreach program activities affect the knowledge, attitudes, and behaviors of its target audiences, and link these results to broader goals, it may wish to consider a tool such as a logic model—as the agency used in its Research and Technology strategic plan—to clearly outline those links. In previous work on educational and outreach, we used a logic model to describe the program components—inputs and outputs—and how these support the desired results, thus explaining the strategy—or logic—by which programs are expected to achieve their goals. Figure 3 graphically depicts this model and its components. The logic model illustrates how program activities such as distributing informational pamphlets contribute to the results of a program, such as increasing safety-belt usage, and to an ultimate goal, such as reducing fatalities. In addition, program managers should consider the effect of external factors when defining outcomes and recognize that the correlation between activities and desired outcomes may not be as direct as one would like. Without a logic model or similar tool to clearly describe the links between education and outreach programs and broader goals, FMCSA may have difficulty designing and evaluating effective programs. FMCSA has begun some evaluations of its education and outreach activities, including the New Entrant program, and plans to use contractors to evaluate some programs. Surveys and studies are the primary means by which FMCSA or its contractor will evaluate the programs. However, some gaps in evaluation plans remain, especially for the largest of the programs we reviewed. Specifically, FMCSA has not evaluated whether the Education and Technical Assistance Package provided to new entrants and the safety audits conducted under the New Entrant program effectively communicated information to new entrants, making it difficult to determine the impact of that program. A list of FMCSA’s evaluations and planned evaluations are provided in figure 4. For each program, FMCSA’s evaluation efforts are described as follows. FMCSA evaluates the performance of the New Entrant program primarily by tracking the number of safety audits conducted of new motor carriers. The agency has a goal of conducting a certain number of safety audits per year, safety audits of 80 percent of new entrants within the first 6 months they are in business, and 100 percent of all new entrants within their first 18 months of operation. FMCSA sets employee performance goals for division office managers for conducting a particular number of safety audits of new entrants. According to FMCSA officials, agency personnel conducted over 9,400 safety audits in fiscal year 2005, exceeding their goal of 8,000 FMCSA completed safety audits. In addition, by April 2005, 52,000 safety audits had been conducted since the program began, and the number conducted each quarter has been generally increasing. However, FMCSA officials acknowledge that there is a safety audit backlog in some states, and the goals for auditing new entrants in the first 6 months, and the first 18 months, are not being met. The number of safety audits reflects the number of new registrants that were personally informed about the carrier requirements. Using registration data on the number of new entrants, the agency can estimate the number of new entrants needing a safety audit and agency progress toward the program goal. There are about 40,000 new entrants registered per year, but officials told us that about 40 percent drop out or discontinue business within the first year of operations. By tracking the number of safety audits conducted, FMCSA seeks to determine the extent to which it has provided information to its targeted audience of new drivers. FMCSA has no process to measure whether new entrants’ awareness of safe trucking practices was raised, which ideally would be a consequence of an effective education and outreach effort. Specifically, although FMCSA plans to evaluate the New Entrant program in 2008, it has no current plan to determine how FMCSA’s safety requirement information, such as its Education and Technical Assistance Package and its safety audits, affect new entrants’ (1) awareness of FMCSA requirements, such as drug and alcohol testing requirements or (2) their motivation to change behaviors that might lead to safer operations. The New Entrant program manager indicated that since the program is new and still evolving, the outreach campaign does not have much historical data to evaluate. Although the program has not been in existence very long, FMCSA officials stated that the agency has conducted one study of the New Entrant program and has plans to conduct another. One preliminary analysis by FMCSA compared the crash rates of new motor carriers registering with FMCSA during 2002 to the crash rates of over 1,000 new motor carriers that registered with FMCSA and had a safety audit during the first 6 months of 2003. This analysis showed that there was little difference between the crash rates of these two groups. FMCSA officials said that they are planning a study to be conducted after planned changes to the safety audit, discussed later in this report, have been in place for a period of time. Officials told us that the study planned to begin in 2008 will examine carriers’ safety records over time, although they have not yet developed an evaluation plan. We have previously reported on the difficulties of assessing education and outreach programs such as the New Entrant program. We have also reported, however, that federal agencies are expected to demonstrate how their activities contribute to achieving agency or governmentwide goals. Additionally, the guidance that FMCSA uses when developing evaluation efforts for education and outreach programs—a booklet entitled The Art of Appropriate Evaluation —published by NHTSA—emphasizes that evaluations should include measurements of changes in awareness, knowledge, and behavior. If FMCSA’s planned evaluation of the New Entrant program does not consider whether the program is actually succeeding in effectively conveying the educational material on safe trucking practices, FMCSA will be unable to clearly determine the program’s impact. Or, if safety audits are having little impact, as suggested by FMCSA’s preliminary analysis, the agency will not be able to determine whether it is due to (1) the educational material provided, (2) how the information is conveyed during the safety audits, (3) the lack of follow-up on the safety audit results, or (4) other factors. Additionally, it will be difficult to demonstrate to Congress that this program is having its desired effect. This program has not yet begun. FMCSA officials plan to award a contract by the end of 2005 to develop, implement, and evaluate the Non-Entrant program. The contractor is to test the effectiveness of the outreach materials while they are being developed to determine if they will reach the intended audience and effectively changed knowledge and attitudes. Development of the materials will include surveying the targeted audience and conducting outreach through truck leasing and sales firms. FMCSA’s program plan indicated that the contractor will be expected to assess changes in knowledge, attitudes, and behaviors of its target audience. FMCSA officials stated that they plan to evaluate this initiative in several ways once they have finalized its evaluation arrangements. According to FMCSA officials, during program design, FMCSA tested its marketing materials with target audiences, including truck drivers, and found that the marketing material was appropriately targeted. Also, FMCSA is installing software to track the number of Web-site hits on certain of its Web pages that contain important educational elements of the Safety Belt campaign. This system will enable FMCSA to determine how many times its information has been viewed, but it will not enable FMCSA officials to know whether that information was effectively conveyed to its target audience or changed their attitudes about wearing safety belts. Nonetheless, FMCSA officials believe that this software will enable program managers to monitor and determine which Web-site pages are viewed, and therefore are more effective, based on the number of hits and the length of time at each site. Furthermore, FMCSA is considering focus groups and surveys with carriers on how safety belt marketing material affected their awareness of the importance of using safety belts. In addition, FMCSA has an agreement with NHTSA for two efforts that could measure progress toward the expected outcome of this initiative—a 10-percent increase in use of safety belts from 2006 through 2009. The first effort is a study, expected to be completed in December 2005, which updates a 2003 commercial motor vehicle study that reported that 48 percent of commercial drivers wear safety belts. The new study will update information on the percentage of commercial drivers that wear safety belts by replicating the methodology used in the 2003 study, according to FMCSA officials. The second effort will investigate the feasibility of annually collecting data on safety-belt use by commercial drivers through the National Occupant Protection Usage Study (NOPUS) sponsored by NHTSA. Although these efforts will measure safety-belt use, they will not evaluate whether the education and outreach program in particular contributed to changes in the number of individuals wearing safety belts. FMCSA has a Web-based, pop-up survey for its Web site which, although focusing on consumer satisfaction in general, gives FMCSA some ideas about Web-site usage and exposure to the information contained on the site. The survey randomly selects visitors to the Motor Coach Web site and asks them to answer a series of questions concerning the accuracy, quality, convenience, and ability of finding needed information. The survey also asks visitors about their overall satisfaction with the Web site. Although these questions may be generally helpful to rate the Web site experience, more specific questions on the Motor Coach program information have not been incorporated. The pop-up survey has flexibility for use in other FMCSA program Web pages, and FMCSA has plans to use Web-based usage tracking as an additional tool in the future. However, with only a small budget—$150,000 in fiscal year 2005—for the Motor Coach program, it is reasonable to expect that FMCSA would balance the need for this type of evaluation information with its other education and outreach program priorities. As of October 2005, FMCSA officials also had plans to award a contract for contractor support to conduct surveys, focus groups, and interviews to evaluate its education and outreach programs, including obtaining Office of Management and Budget (OMB) clearance for these activities. FMCSA officials told us that the motor coach program would be evaluated under this contract. FMCSA has identified at least two approaches to evaluate the effectiveness of the outreach efforts for the household goods outreach program. One effort is to survey recipients who have received the Ready to Move - Tips for a Successful Interstate Move booklet from GSA. As of May 2005, 703 surveys had been distributed to determine whether the recipients believed that the booklet would change their intentions about which household goods carrier they might select. The survey asks questions regarding the effectiveness of the booklet and its information content. Also, FMCSA officials stated that the Household Goods program would be evaluated under the survey support contract discussed in the preceding section on the Motor Coach program. An FMCSA official whom we spoke with indicated that the elderly may be particularly targeted by unscrupulous movers. Recent data released by the Census Bureau indicate that only about 40 percent of people age 55 years or older have computers with Internet access in their homes. Although FMCSA has an available hard-copy brochure, the Web-based outreach efforts may not be fully reaching this population. FMCSA’s planned evaluations of the Household Goods program do not address the effectiveness of FMCSA’s Web-based outreach in reaching this population. Most of FMCSA’s evaluations of its education and outreach programs have yet to be completed, and therefore little is currently known about the effectiveness of the programs. FMCSA expects the contracts for the evaluation programs to be finalized within months and will receive information from the evaluations once the data are collected and analyzed. Until that time, what is known about program effectiveness is largely based on anecdotal information from stakeholders such as industry associations, as well as a comparison of the education and outreach programs to the design of other successful programs that have been implemented. However, FMCSA’s education and outreach programs and marketing materials appear to follow theories and research on behavior change. Finally, a public-safety group suggests FMCSA followed reasonable approaches in starting its education and outreach efforts, however, it would like to see more quantitative evaluation in the future to help FMCSA refine its programs. Studies by us, other agencies, and academics show that education and outreach programs, such as those undertaken by FMCSA, can be effective in creating awareness of a problem or issue and influencing some change in behavior. One theory—the theory of reasoned action—explains the relationship between attitudes, beliefs, intentions, and behavior. In short, people are more likely to change their behavior if they believe the change is valued—or that not changing their behavior has negative consequences—and if they have the necessary opportunity to change the behavior. For example, one antismoking advertising campaign emphasized the negative effects of smoking on a smoker’s health. Smokers became aware of the negative consequences, and 26 percent of respondents indicated that they were more likely to quit smoking based on the awareness of the negative consequences of smoking. Another study on the effects of sun exposure showed that countering the mistaken belief that sun tanning had a positive effect reduced the percentage of individuals who habitually lie in the sun. These studies demonstrate that emphasizing the consequences of a negative behavior to individuals can result in changes to attitudes and behavior, without applying any specific sanctions. Based on the theory of reasoned action, education and outreach can have a positive impact through FMCSA’s two programs that have a goal of influencing consumers to make appropriate choices of motor carriers. FMCSA’s Motor Coach and Household Goods Outreach programs are targeted toward consumers and do not have regulations governing the consumers’ behavior, thus, according to the theory of reasoned action, gaining knowledge about the positive consequences of choosing a safe carrier will affect the individuals’ attitude toward this behavior. The change in attitude will directly affect intention and the subsequent behavior. Thus, education and outreach campaigns can provide an individual under these circumstances with the information that is needed to make a more educated choice. For example, the Motor Coach program follows this approach by targeting consumers and conveying the negative consequences of choosing an unsafe carrier. In the case of the Household Goods program, the campaign conveys the positive consequences of choosing a reputable carrier in the Households Goods program, although the elderly, a group potentially susceptible to unscrupulous carriers, may require a different delivery method other than FMCSA’s Web site to receive this message. Independent studies as well as our prior work indicate that the level of enforcement in safety related programs is positively correlated with safety-belt use, especially when coupled with public-awareness efforts. According to the general deterrence theory, individuals must be exposed to law enforcement or receive information about legal ramifications of their actions before they can be deterred. The perception or expectation of certain sanctions results in attempts to avoid committing the offense. Studies show this theory is an effective model for strengthening the change in behavior resulting from safety programs. Thus, this link to enforcement would apply to FMCSA’s three education and outreach programs—Safety Belt, Non-Entrant and New Entrant—that have existing regulations that address their target audience’s behavior. One study, conducted by the National Safety Council, on the state of South Carolina’s safety-belt program, showed that while media campaigns immediately increased awareness of the importance of safety belts, the largest increase in safety belt-use occurred after officers enforced the requirement to wear safety belts. Additionally, a 2002 study by NHTSA included data from the state of Washington, which showed that the baseline percentage of individuals wearing safety belts (81 percent) increased slightly with advertising over a 4-week period to 82 percent. Ultimately with the addition of enforcement efforts, the percentage of those who wear safety belts increased to 90 percent within 2 months. The same study showed similar results in Texas, where the effects of advertising alone raised the safety belt percentage slightly, but the combination of advertising with enforcement caused the number to increase another 6 percent, from the 80-percent baseline. These studies suggest that although publicity may increase the awareness of the need to wear safety belts, some people are unlikely to act on that awareness unless the enforcement component of the campaign was closely integrated with the publicity campaign. The results of these studies suggest that education and outreach alone can increase awareness and encourage the intended behaviors for FMCSA’s education and outreach programs, but that enforcement can strengthen the result of these efforts. Three of FMCSA’s programs we examined, the New Entrant, Non-Entrant and Commercial Motor Vehicle Safety Belt programs have regulations, such as a requirement for motor carrier drivers to wear safety belts, affecting the target audiences’ (motor carrier industry) behavior. This provides the opportunity to use both enforcement and education in programs with the goal of improving the safety behavior of the motor carrier industry. The following sections describe how FMCSA has indicated it is using or will be using enforcement along with education and outreach in these programs. Enforcement has been only a minor component of the New Entrant program, but FMCSA is currently strengthening the criteria for passing New Entrant safety audits because of concerns of its effectiveness, based on analysis and anecdotal information indicating that new entrants that have been notified of the safety audit requirements may not be operating more safely. In addition, even if a new entrant passes the safety audit, some carriers are not yet following important safety requirements, such as having a drug and alcohol testing program, at the time the safety audit was conducted. This change should establish a better link to enforcement, in keeping with behavioral research, in that it will be more difficult for new entrants to retain their operating status unless they are following the safety requirements at the time of the audit. The safety audit has been considered an educational unit, with a strong emphasis on training new entrants about the requirements to operate in interstate commerce. For example, the federal register notice that established the new entrant safety assurance process indicated that the new entrant process is intended to “improve the safety performance of new entrants by providing educational and technical assistance” and required “minimum requirements to new entrant motor carriers to ensure that they are knowledgeable about applicable federal motor carrier safety standards.” There are few sanctions for carriers who fail critical areas of the safety audit, although a new entrant might be subject to a more stringent and detailed review—a compliance review—under certain circumstances. However, the compliance review process occurs less often with new motor carriers than experienced carriers. Although FMCSA’s requirements indicate that the objectives of the safety audit are to educate the carrier and determine areas where the carrier’s compliance might be deficient, in practical application almost all new entrants have passed the safety audit, and little is done to follow-up to ensure that new motor carriers correct deficiencies identified in the safety audits. A new entrant can fail two of six sections and still pass the audit. Safety audit sections include areas such as driver qualifications, business operations, and maintenance records. Over a 2-year period ending in April 2005, there were 51,681 carriers that passed the safety audit, and only 372 failed, producing a 99.3 percent passing rate for new entrant motor carriers. Those who fail the audit must take corrective action within 45 days if they are a passenger or hazardous material carrier, or 60 days for all other carriers, or they will lose operating status. Furthermore, for those carriers who passed the audit despite failing multiple sections of it, FMCSA currently has no mechanism to ensure the failed sections of the audit have been corrected before the carrier achieves permanent status. Although FMCSA sends letters to new entrants about the sections in which they are not in compliance with motor carrier regulations; beyond the letter and agency follow-up to obtain proof of insurance, the agency or its representatives do not determine if the new entrants have in fact corrected deficiencies identified during the safety audits. Our analysis of critical factors within the safety audit questions revealed that over the same 2-year period, about 40 percent of the carriers failed the “driver” section of the audit, despite passing the audit overall. This section includes critical questions, such as whether a carrier’s driver qualifications have been reviewed and whether the carrier's drivers are covered by a drug and alcohol testing program. Specifically, we found that about 60 percent of the new entrant carriers that passed the safety audit had failed the requirement to check qualifications of drivers prior to hiring them. Checking qualifications means that before hiring a driver a motor carrier company obtains and reviews information on an applicant’s driving history, including accident and ticket information. In addition, we found that for the most part, between 27 and 29 percent of carriers who passed the safety audit had not maintained maintenance files for requested vehicles. The lack of improvement over the 2-year period illustrates the importance of determining if the Education and Technical Assistance Package, which the carriers received prior to the audit, should be changed to improve these results. FMCSA officials agreed that the safety audit may not be improving safety behavior of new entrants, and the agency is already tightening these requirements. Currently, there is no one question or section of the audit that can cause a new entrant to fail. FMCSA officials informed us that the new federal register notice laying out stricter pass requirements—expected to be issued shortly—identify eight critical questions that individually could lead to failing the safety audit, thus providing more incentive for carriers to comply with new entrant requirements. For example, carriers would automatically fail if they do not employ a drug and alcohol program, have insurance, or have the proper maintenance requirements. Carriers who fail the safety audit and do not correct the deficiencies could lose their operating status and would be required to correct deficiencies before their operating status is reinstated. Similarly to the New Entrant program, there has been limited enforcement associated with the Safety Belt program to date. The finding that 48 percent of commercial drivers wear safety belts, compared with 80 percent of passenger vehicle drivers, suggests that commercial motor vehicle operators have thus far been less receptive to messages in the market place regarding the importance of wearing safety belts while on duty. In light of the studies we reviewed, an enforcement element might help strengthen education and outreach programs’ attempts to increase awareness of the risk to drivers of not wearing safety belts and encourage commercial drivers to wear them. It is reasonable to assume that commercial motor vehicle operators, similar to private operators, would be educated and informed from the materials presented in the campaign but that some would not act on the information unless there was some risk of sanction. The approved plan for FMCSA’s safety belt initiatives includes a third phase of implementation that calls for increased enforcement. Although the plan does not specify how FMCSA will enforce federal regulations regarding wearing safety belts and how FMCSA will work with state agencies to carry out enforcement—as this program develops, strengthening the enforcement link may increase the effectiveness of the safety belt education and outreach efforts. FMCSA officials indicated that this enforcement effort had begun and included fiscal year 2005 grants to state agencies that can be used for safety belt education and enforcement. FMCSA’s program plan for the Non-Entrant program indicates it will have some enforcement component as it is developed; however, specific information on enforcement has not yet been developed. Literature on safety programs and experience with similar programs indicates the Non-Entrant program might benefit from enforcement after non-entrants are exposed to the message about the importance of registering for a DOT license when they buy or rent a vehicle for operation. Overall, the industry associations’ view is that FMCSA is doing several positive things in its education and outreach activities. Officials of the seven industry associations whom we spoke with had regular contact with FMCSA officials and were familiar with the education and outreach programs under this review. Many association officials cited direct contact with FMCSA on industry issues on a regular basis and said that their partnerships with FMCSA on education and outreach activities were productive. Specifically, an official of one moving industry association told us that it shared data with FMCSA regarding complaints against carriers and often refers complainants to FMCSA for further information. Officials representing a consortium of primarily state agencies involved in motor carrier safety, to which FMCSA belongs, told us that the support FMCSA was able to provide was important to the implementation of the Safety Belt program. Lastly, some association officials said that FMCSA was doing a good job with the resources it has, but others said FMCSA could do more with its education and outreach efforts if the agency had more resources. However, one of the two public safety groups we contacted believed that although the education and outreach programs were based on good ideas, FMCSA needs to do more quantitative evaluations of these programs. Both public safety groups also felt that FMCSA should have stronger enforcement. For example, one group indicated that research has shown enforcement to increase the effectiveness of education programs and thus, FMCSA should focus more on compliance and enforcement. Finally, although industry associations emphasized the positive professional relationships they have with FMCSA officials, a public safety group indicated that it is wary about FMCSA’s tendency to manage programs in alliance with industry partners, since the group believes it might affect the agency’s objectivity. Uncertainty about how FMCSA’s education and outreach activities link to broader program and agency goals and about FMCSA’s New Entrant program have emerged from our study. FMCSA officials have stated that the agency’s education and outreach programs support department objectives and that they have used a logic model to show this in the agency’s performance budgets. However, in our review of agency documents, we found that except for the Safety Belt program, FMCSA has not established clear links between the education and outreach programs and agency and departmental goals. For example, the conference report accompanying the DOT appropriations bill for fiscal year 2005 requested FMCSA report to the House and Senate Committees on Appropriations “a goal, message, and coherent and explicit program strategy that clearly and directly link FMCSA’s outreach and education program initiatives to each program’s goals” because of its concerns over how FMCSA’s education and outreach activities support agency goals. In the October 2005 response, FMCSA stated that its education and outreach programs support DOT strategic objectives. The report, however, did not clearly set out the links between changing attitudes and behaviors that would describe how these programs contribute to objectives. Such a step would be welcome, for without an explanation of that link, FMCSA cannot transparently demonstrate to stakeholders, including congressional oversight authorities that the programs do, in fact, contribute to these broader goals. Furthermore, without a clear statement of how these programs are intended to support these goals, it is difficult to design evaluations to determine if these programs are performing as intended or if changes are needed in program design. FMCSA has used a type of a logic model for its Research and Technology program, making it clear to all interested parties how its research activities help the agency achieve its goals. While FMCSA officials say they have used a logic model for its education and outreach program, this is not evident in agency budget and program documents. The use of a logic model, similar to the one used by the agency’s Research and Technology group, in planning, budgeting or program for education and outreach programs may be beneficial. Second, the absence of a plan to evaluate how well the information provided to motor carriers as part of the New Entrant program was understood and if the information raised awareness and helped promote safer behavior also emerges as a concern. An FMCSA official stated that the New Entrant program is still new and has limited data to evaluate. Further, changes to the program are planned to raise the threshold for passing a safety audit. Thus, FMCSA is planning on starting an evaluation of the New Entrant program in 2008. However, without an evaluation of the effect of education on new entrants’ knowledge about safety requirement, FMCSA is losing an opportunity to obtain information about the effectiveness of the educational information conveyed through the program. In addition, FMCSA cannot know whether its safety audits, which composed almost 7.5 percent of FMCSA’s 2005 funding, are having the desired effect; whether the Education and Technical Assistance Package and the information conveyed during the safety audits might require some changes to improve their effectiveness; and whether the safety audits themselves might be achieving their purpose of improving motor carrier safety. FMCSA has two opportunities upcoming to obtain information on the effectiveness of the educational materials and safety audits. First, it could use its planned contract for evaluation of education and outreach activities to develop and conduct such an evaluation. Second, FMCSA could include this evaluation in its plans to evaluate the New Entrant program beginning in 2008. To better demonstrate how FMCSA education and outreach programs contribute to achieving agency goals, we recommend that the Secretary of Transportation direct the Administrator of FMCSA to take the following two actions: Ensure that the agency describes and documents how education and outreach program activities link to and support broader program and agency goals in a planning, program, or budget document that is available to the public, and Evaluate the effectiveness of the education and outreach of the New Entrant program, assessing the extent to which the Education and Technical Assistance Package and safety audits are helping new carriers learn and understand FMCSA requirements. We provided a draft of this report to the Department of Transportation for its review and comment. FMCSA officials, through the DOT liaison, provided oral comments on a draft of this report. The officials did not agree with GAO’s finding that FMCSA has not linked its education efforts with its overall goals. FMCSA officials state they have linked their education and outreach programs to agency goals at a high level in the agency’s performance budgets. In addition, the officials state that they used a logic model to make this link. We included FMCSA’s view that it has linked its education and outreach programs to agency goals in our report. However, we did not find this link evident for most of the education and outreach programs. For example, Congress requested FMCSA to report on the link between its education and outreach program initiatives to each program’s goal. FMCSA’s report to Congress described the program activities; however, it did not describe the link between the activities and FMCSA’s goals. In addition, although FMCSA officials stated that they have used a logic model in their performance budget to link education and outreach programs to agency goals, the discussion implied a link, but does not describe how FMCSA’s education and outreach activities are intended to change target audience’s attitudes and behaviors that would contribute to meeting agency goals. However, in FMCSA’s fiscal year 2006 performance budget, we did find and report on a relatively clear discussion of how the Commercial Motor Vehicle Safety Belt program supported safety objectives such as reduced fatalities. In addition, DOT and FMCSA officials were concerned that our recommendation on documenting the link between education programs and agency goals was too prescriptive. We recommended that FMCSA expedite the process of describing and documenting how its education and outreach programs link to and support overall agency goals in strategic planning and budget documents. DOT officials said that including this specific information on relatively small dollar projects—such as the Motor Coach program—was not appropriate in the DOT strategic plan. In addition, DOT officials told us that it is not appropriate to include this level of performance information in budget requests. Since the strategic plan and the performance budget are not the only opportunities to detail program linkage to broad agency goals, we clarified our recommendation to provide FMCSA with more flexibility in identifying an appropriate planning, program, or budgeting document in which to describe and document the link between program activities and goals, so that external stakeholders, including congressional oversight authorities, can understand how these programs are supporting agency goals. Second, agency officials suggested that it is too early to evaluate the New Entrant program, as we recommended; and they have not developed evaluation plans because they do not plan on beginning an evaluation until 2008. However, after over 2 years of implementation and 52,000 safety audits, FMCSA is losing an opportunity to conduct an evaluation of the educational component of the New Entrant program, including the Educational and Technical Assistance Package for new entrants and the information conveyed during safety audits to determine if new motor carriers increase their knowledge of motor carrier requirements and change their behavior to better comply with these requirements. The lack of such an evaluation is a missed opportunity for FMCSA to demonstrate the effectiveness of its educational information and safety audits, which represented almost 7.5 percent of its funding in fiscal year 2005. In addition, we continue to believe this is an important component of any evaluation FMCSA develops to determine the effectiveness of the New Entrant program. Therefore, we retained our recommendation to evaluate the New Entrant program and conclude that there are two opportunities for doing so—either through contracts for evaluations of education and outreach programs or through the evaluation FMCSA is planning to start in 2008. Finally, agency officials provided technical comments, which we incorporated as appropriate. We are sending copies of this report to the Honorable Norman Mineta, Secretary of Transportation. We will also make copies available to others upon request. In addition, the report will be available at no charge on the GAO Web site at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at siggerudk@gao.gov or (202) 512-2834. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. Major contributors to this report are listed in appendix III. To address our first objective and describe the scope and nature of the Federal Motor Carrier Safety Administration’s (FMCSA) education and outreach programs, we reviewed program documentation and interviewed FMCSA officials at agency headquarters, its Eastern Service Center in Maryland, and its New Jersey division office. We selected this division office and service center because they were significantly involved in all four of the currently active education and outreach programs we reviewed, which are the New Entrant, Commercial Safety Belt, Motor Coach Outreach, and Household Goods Outreach programs. We obtained and reviewed information on FMCSA’s plans to initiate an additional program targeted at non-entrant carriers. Further, to determine how these programs relate to FMCSA’s goals, we reviewed its strategic planning and performance budgeting documents and how these education and outreach programs were portrayed in the current fiscal year 2006 documents. We also discussed the performance budgeting process with an FMCSA senior budget analyst to identify the agency’s plans for the fiscal year 2007 and 2008 performance budgets. We also obtained and reviewed the Department of Transportation’s (DOT) strategic plan, which FMCSA refers to in its performance budget. In addition, we obtained the strategic plan for FMCSA’s Research and Technology group. We reviewed this plan to identify what one group within FMCSA uses for its strategic and program planning. To identify other tools used for describing how program activities relate to agency goals, we reviewed our previous work and identified a program logic model, originally developed by the University of Wisconsin Cooperative Extension, as a model that FMCSA could use to clearly describe its education and outreach programs. To address our second objective—identifying the extent to which FMCSA has evaluated its education and outreach programs, we discussed the status of these evaluation efforts with FMCSA officials and reviewed proposals and statements of work for current and planned contracts to carry out program evaluations. We also assessed the usefulness of FMCSA’s ongoing or planned program evaluations to determine if the ealuations would address how program activities affected target audiences’ attitudes, knowledge, and behavior. We did not thoroughly assess the evaluation design. We took several approaches to assess what is currently known about the effectiveness of programs like FMCSA’s education and outreach programs. Finally, to identify what is known about the effectiveness of the programs, we conducted a literature review of media based campaigns and reviewed the structure and evaluation of these campaigns as well as behavioral theories used in designing the campaigns. We selected two behavioral theories that were directly applicable to the programs under this review— the theory of reasoned action and the general deterrence theory. We then directly compared the characteristics of the two theories with the characteristics of FMCSA’s programs. Additionally, through our literature search, we also identified additional research that discusses the importance of the relationship between education and enforcement in changing behavior. We used this research to help assess the potential for using enforcement to augment FMCSA’s education and outreach efforts. We obtained aggregated data from safety audit results from FMCSA’s Motor Carrier Management Information System (MCMIS) to examine results for individual elements within a safety audit. We used these data to identify the failure rates for key elements of the safety audit. We conducted a reliability assessment of MCMIS data by (1) reviewing documentation related to system development, (2) interviewing knowledgeable agency officials, and (3) reviewing FMCSA’s internal controls, and we determined the data was sufficiently reliable for our use. Finally, we interviewed officials from several motor carrier, law enforcement, and safety organizations, including the American Trucking Associations; the National Private Truck Council; the American Bus Association; the United Motorcoach Association; the American Moving and Storage Association; the Commercial Vehicle Safety Alliance; Advocates for Highway and Auto Safety; and Public Citizen for their perspective of the effectiveness of FMCSA’s education and outreach efforts. In accordance with congressional direction, we did not extend our review to include “Share the Road Safely” which is targeted to passenger car drivers. In fiscal year 2004, Congress transferred funding for the program from FMCSA the National Highway Traffic Safety Administration (NHTSA). Funding for this program is being returned to FMCSA in fiscal year 2006. In fiscal years 2004 and 2005, Congress provided FMCSA with funding for another outreach effort entitled “Safety Is Good Business.” The SAFETEA- LU requires GAO to review this program by June of 2006. Our work was conducted in accordance with generally accepted government auditing standards, from December 2004 through October 2005. The following provides a more detailed discussion of the five education and outreach programs we reviewed. The New Entrant program is intended to ensure that new motor carriers are knowledgeable about federal motor carrier safety standards and applicable hazardous material regulations. The program is focused on the safety audits conducted through site visits to the new carriers’ offices by personnel from the state law enforcement agency, the FMCSA division office, or a FMCSA contractor. In 1998, a FMCSA commissioned study found that new entrants to the motor carrier industry have a substantially higher level of safety regulation violations than more experienced carriers. Other, earlier research had also indicated that new entrant carriers had crash rates higher than more experienced carriers. The Motor Carrier Safety Improvement Act, which established FMCSA, also directed the Secretary of Transportation to establish regulations specifying minimum requirements for applicant motor carriers seeking federal interstate operating authority and to require that new entrants undergo a safety audit within the first 18 months of operations. Effective January 1, 2003, FMCSA established its New Entrant program to improve the safety of new entrant motor carriers by providing them with educational and technical assistance as they begin their businesses. Under FMCSA’s New Entrant program, any new motor carrier that registers with DOT is designated a “new entrant” and will be subject to a new entrant safety audit and increased roadside performance monitoring for an 18-month period. The safety audit’s purpose is to provide educational and technical assistance to the new entrant by reviewing the carrier’s compliance with FMCSA requirements, including driver qualifications, driver records of duty status, vehicle maintenance records, accident registers, and controlled substances and alcohol use and testing requirements. Education and technical assistance information is also available to new entrants through FMCSA’s Web site. A carrier that passes the safety audit is informed of any deficiencies and what it must do to be in compliance. A carrier that fails the safety audit has 60 days to take necessary corrective action, and ultimately if the deficiencies are not corrected, its registration can be revoked. At the end of the 18-month period, if the carrier has passed its safety audit, does not have an unsatisfactory safety rating, and is not subject to an “out of service” order or any notice to remedy safety management controls, the “new entrant” designation will be removed from its registration, and it will be monitored like any other carrier. In most states, new entrant safety audits have been done by both state law enforcement agency and FMCSA division office personnel. Contractors conduct safety audits in four states (Florida, Maine, Oregon, and Wyoming) that are presently doing few or none of their own. When the New Entrant program began, FMCSA staff performed most of the safety audits, but states now carry out the majority of these audits, as indicated by figure 5. FMCSA’s Non-Entrant program will target motor carriers that operate interstate but have not registered with DOT. The purpose of the program will be to raise awareness of registration requirements, and to increase the number of registrations among these non-entrant motor carriers. This program has not yet begun, but FMCSA officials told us it will include information disseminated through truck sales and leasing firms to non- entrants. The number of non-entrants is unknown, but a FMCSA official estimates it may be in the tens of thousands. FMCSA officials whom we spoke with believe that non-entrants may represent an even higher level of risk than new entrants, because they do not get exposure to FMCSA safety information through registration. The officials believe that, in general, motor carriers that do not register with DOT are unaware of the requirement to do so, rather than intentionally avoiding registering. FMCSA will be awarding a contract for the development of the Non-Entrant program; the contract is expected to include a requirement to identify the target population of non-entrants, develop informational material, with follow-up monitoring of non-entrants that are identified. FMCSA’s Commercial Safety Belt Program is intended to improve upon the low rate of safety-belt use by commercial truck drivers. In 2003, a study commissioned by FMCSA reported that only 48 percent of commercial truck drivers wear safety belts, compared with 80 percent of passenger car drivers wearing safety belts. In December 2003, FMCSA established the Commercial Motor Vehicle Safety Belt Partnership with 16 other organizations, including trucking industry and law enforcement associations, commercial drivers associations, and safety organizations, with the goal of increasing commercial safety-belt use by commercial motor vehicle drivers. With its safety belt partners, FMCSA produces materials including posters, bumper stickers, and brochures for distribution at trucking industry shows and events, and through law enforcement campaigns. In 2004, as part of the Partnership, FMCSA helped distribute informational brochures developed by the Commercial Vehicle Safety Alliance (CVSA). In 2005, FMCSA introduced and distributed new materials with the theme “Be Ready. Be Buckled.” for distribution at national and regional trucking industry shows and events, and through the CVSA-sponsored Road Check 2005. This was a 3-day truck and motor coach inspection effort in June 2005 that involved over 9,900 inspectors, conducting more than 60,000 vehicle inspections in the United States, Canada, and Mexico. Through the Commercial Motor Vehicle Safety Belt Partnership, FMCSA also communicates its message through information displayed at truck stops and satellite radio programming oriented to truck drivers. FMCSA’s objective for the motor coach industry is to reduce injuries and save lives through increased compliance with safety regulations and promotion of safe operations by motor coach owners and operators. FMCSA’s approach is two-fold—(1) reaching out to the industry to inform motor coach companies of regulations and provide safety advice and (2) providing consumers with information to help them in selecting and hiring motor coach services. FMCSA’s motor coach outreach activities include communication with the carriers through printed material and participation in trade shows, and Web-based information for the public. For the purposes of our review, we focused on FMCSA’s outreach to members of the public that hire motor coach services. Motor coaches have been one of the safest forms of commercial transportation, but an increase in motor coach occupant fatalities since 2001 led FMCSA to increase its focus on motor coach companies. In 2002, FMCSA developed “Moving Kids Safely” a toolkit of reference materials targeted to school districts and others who contract for bus services. In 2004, FMCSA introduced a motor coach safety Web site within the FMCSA Web site, which provides guidance to consumers on selecting a motor coach company and allows consumers to access safety data on specific interstate motor coach companies. FMCSA’s most direct outreach to motor coach companies is through the agency’s Technical Assistance Group (TAG), consisting of about 12 FMCSA staff with motor coach expertise, primarily in FMCSA’s field offices. TAG personnel staff information booths and give presentations at industry conventions and events, and serve as agency contact points for carriers. Also, a FMCSA contractor developed a series of brochures aimed at motor coach companies that describe guidelines and regulations for motor coach companies (for instance licensing and insurance, hours of service, inspection and maintenance, and specific requirements for operators of school buses). In 2004 and 2005, FMCSA also conducted mass mailings to motor coach companies, advising them of their reporting requirements under the Americans with Disabilities Act. Most households need to hire moving services infrequently, and therefore consumers may be unaware of what their rights and protections are in hiring a commercial mover. The overriding objective of FMCSA’s Household Goods program is to help consumers make better informed choices in selecting and negotiating with a moving company, primarily through printed material distributed through moving companies and other sources, as well as providing information through its Web site. FMCSA also maintains a complaint hotline to receive complaints about movers from individuals. The ICC Termination Act of 1995 transferred federal responsibilities for protecting consumers who use commercial moving companies for interstate moves to DOT; and the Motor Carrier Safety Improvement Act granted FMCSA continuing oversight responsibility over household goods movers. However, the act did not provide FMCSA with the authority to intercede on behalf of individuals seeking reimbursement of moving company charges or recovery of their goods. FMCSA officials whom we spoke with acknowledge that when the agency was established in January 2000, its mandate was commercial vehicle accidents and fatalities and, as a result, the household goods industry received little attention. Also, in 2001, we reported that complaints about movers were on the rise and that DOT’s lack of action had created a vacuum that allowed unscrupulous carriers to take advantage of consumers. At that time, our report noted that FMCSA had just begun planning to increase both consumer education and enforcement effort of household goods carriers. FMCSA distributes three consumer-oriented publications. Interstate movers are required by law to provide a copy of one FMCSA publication— “Your Rights and Responsibilities When You Move” to prospective customers. A second brochure, “Ready to Move? Tips for a Successful Interstate Move” is available through the General Services Administration’s Pueblo, Colorado Information Center. A third brochure “Protect your Memories…Protect Yourself from Moving Fraud” and the “Your Rights and Responsibilities” publications are available on FMCSA’s Web site. In March 2005, FMCSA awarded a contract for the development of a household goods education and outreach program. FMCSA and the contractor have developed a new outreach theme: “Protect Your Memories, Your Money, Your Move”; and the contractor will be developing brochures, presentations that can be delivered by FMCSA staff, and other materials. In June 2005, FMCSA launched its redesigned household goods Web page, designed around the “Protect Your Memories…” theme, which incorporates new content, including a checklist for moving and information on protecting against moving fraud. In November 2005, FMCSA officials informed us that they were in discussions with the U.S. Postal Service to have the Postal Service Web site provide a link to FMCSA’s Web site when registering a change of address with the Postal Service. FMCSA officials told us they do not arbitrate complaints that the agency receives, but they maintain a national consumer complaint database that is used to identify problem movers for investigation by FMCSA field staff. Our 2001 report on DOT and FMCSA oversight of the household goods industry recommended that FMCSA make information on the number and general nature of complaints against carriers be made available to the public. FMCSA officials whom we spoke with said the agency intends to do this, but Privacy Act and other issues need to be resolved beforehand. In addition to the person named above, Cathy Colwell, Assistant Director; Colin Fallon; Eric Fielding; Donald Kittler; Sara Ann Moessbauer; and Elaine Vaurio also made major contributions to this report.

Summary: The Federal Motor Carrier Safety Administration (FMCSA) is responsible for improving commercial vehicle safety and uses education and outreach as part of its efforts. The House report accompanying the fiscal year 2005 Department of Transportation (DOT) appropriations bill asked GAO to report on FMCSA's education and outreach programs to the House and Senate Committees on Appropriations. GAO (1) describes FMCSA's education and outreach programs and how they relate to FMCSA's goals (2) identifies the extent to which FMCSA has evaluated its education and outreach programs and (3) describes the extent to which FMCSA's education and outreach programs are effective. FMCSA's education and outreach programs--New Entrant, Non-Entrant, Motor Coach, Safety Belt, and Household Goods' target different audiences, including the motor carrier industry, commercial vehicle drivers, and the public.Total funding for these programs in fiscal year 2005 was $36.3 million; the largest share (about $33 million) went to the New Entrant program, which is designed to inform newly registered motor carriers (new entrants) about motor carrier safety standards and regulations to help them gain compliance with FMCSA requirements. FMCSA uses many approaches, such as direct contact with carriers, media campaigns, distributing printed materials, and establishing Web sites to provide information to target audiences. FMCSA has not described how its education and outreach program activities link expected changes in attitudes and behavior to broader goals, such as DOT's strategic objective of reducing transportation-related fatalities. FMCSA officials state that the education and outreach activities and programs link to agency goals at a high level, but this was not evident from our review, with the exception of the Safety Belt program. FMCSA has used a logic model as a tool in other programs to show the relationship between program activities and broader goals. FMCSA has begun some evaluations of its education and outreach programs, and plans other evaluations of these programs. However, although FMCSA's New Entrant program has existed for over 2 years, FMCSA has no plans to evaluate its New Entrant program until 2008. Thus FMCSA has no information on whether information on its safety requirements, provided through the Education and Technical Assistance package or during New Entrant safety audits--targeted toward truckers newly entering the industry--effectively communicate information to new entrants. This lack of evaluation makes it difficult to determine the impact the education portion of the New Entrant program has on commercial motor vehicle safety. Since FMCSA currently has little information on how its programs have affected attitudes and behavior, it is difficult to determine the effectiveness of FMCSA's effort. However, the designs of two programs appear to follow theories and research regarding media campaigns, which are intended to influence decision making about safety. Research and behavior theory suggest that for some types of programs--such as DOT's Click It or Ticket program, which is designed to increase safety-belt use by passenger car drivers--enforcement linked to education can improve results, and FMCSA has indicated it is linking some education and outreach programs to enforcement efforts, where appropriate. Finally, motor carrier association officials whom we spoke with stated that, in their view, FMCSA is doing some positive things in its education and outreach activities. A public safety group stated that FMCSA followed reasonable approaches in starting its education and outreach efforts; however, they would like to see more information on program effectiveness to help FMCSA refine the programs.