In the early 1990s, federal agencies and Congress recognized that doing business electronically could save resources, streamline the procurement process, and improve the private sector's access to federal contracting opportunities. NASA first used NAIS at its Marshall Space Flight Center in 1994 to post midrange procurement opportunities on the Internet. NAIS services and features expanded rapidly, and the system began to be used NASA-wide in 1995. It was the first agencywide procurement information system on the Internet. With some exceptions, federal agencies must publish notices of competitive contracting opportunities above $25,000 in the CBD. To accommodate timely delivery of notices through a paper-based mail delivery system, the law generally prescribes a 15-day minimum interval between publication of a notice in the CBD and the issuance of a solicitation by an agency. After a solicitation is issued, agencies generally must give contractors a minimum of 30 days to prepare and deliver bids or offers. In support of NASA's test of alternatives to the use of paper documents for communicating business opportunities to prospective contractors, FASA authorized NASA to waive the CBD notice and waiting period requirements for 4 years (until Oct. 13, 1998) and for up to $100 million in procurement life-cycle costs. Congress has supported the use of federal electronic commerce and electronic access to procurement information. Under FASA, this capability was to be provided through the Federal Acquisition Computer Network (FACNET), a governmentwide systems architecture for acquisitions based on electronic data interchange, which is the computer-to-computer exchange of routine business documents using standardized data formats. FACNET implementation proved to be problematic, however, and in 1997, section 850 of the National Defense Authorization Act for Fiscal Year 1998 replaced the preference for FACNET with a more flexible policy that promotes the use of electronic commerce techniques whenever practicable or cost-effective. Section 850 also requires that in implementing electronic commerce, agencies apply standards consistent with nationally and internationally recognized standards that broaden interoperability and ease of use. Agency notices of procurement opportunities must be provided in a form that allows convenient and easy user access through a single governmentwide point of entry. While CBDNet satisfies the requirement with respect to notices, neither it nor other currently available electronic procurement posting systems satisfy the requirement for a single governmentwide point of entry for access to solicitations and other procurement information. The law also requires that the Administrator of the Office of Federal Procurement Policy (OFPP) submit to Congress yearly reports through 2003 on the progress being made in implementing section 850. The reports must include a strategic plan for implementation of governmentwide electronic commerce and an assessment of compliance with the requirement for a single point of entry. Since 1994, NAIS has evolved into a procurement information system with many operational features that are effective, easy to use, and beneficial to NASA and vendors. By eliminating many of the steps in the paper-based process of publicizing contract opportunities and issuing solicitations, NAIS has contributed to a more streamlined agency acquisition process. Feedback from NAIS users, particularly small businesses, has generally been positive. However, the lack of data makes it very difficult to quantify specific benefits of NAIS, such as the amount of time it actually saves. NASA exceeded the $100-million ceiling authorized by FASA for the test. After NAIS implementation, the number of offers per contract and small businesses' share of contract actions and dollars showed some increases. Since July 1994, NAIS has evolved to provide a broad range of procurement-related functions and information. For example, in November 1995, NASA added an on-line procurement reference library. In March 1997, NASA completed the NAIS Electronic Posting System (EPS), which enables procurement staff anywhere in the agency to prepare and post notices and place solicitation files directly onto the NAIS Internet site. In March 1997, NASA also added an e-mail notification service that automatically notifies subscribers of new procurement information available on NAIS. We found that the system's operational features were easy to use and worked as described. Procurement staff demonstrated how they use the EPS to generate, edit, and post notices and place solicitation files directly onto the Internet. The EPS automatically formats each NAIS posting, inserts links to other relevant documents and external information, updates the business opportunities index library, and constructs and feeds messages to the NAIS e-mail notification service. Appendix I provides more information on the development and evolution of the EPS and the approach NASA used to design, develop, and operate the system. By going to the NAIS Internet site, users can identify, at one location, notices of and solicitations for competitive business opportunities and awards over $25,000 at all NASA centers; search for business opportunities by type of product and/or service, NASA center, or posting date; locate and download documents related to a specific procurement; and view summaries of NASA's current contracts (with contractor name, value, obligations, and description) and search current contracts by state, congressional district, business type, or type of product and/or service. The e-mail notification service automatically transmits announcements to subscribers immediately after new procurement information is posted on NAIS. This allows subscribers to track developments in the solicitation cycle and learn of new releases. Subscribers can register free of charge by providing an e-mail address and selecting the types of contracting opportunities in which they have the greatest interest. Thus, instead of having to search for information, subscribers can have it selected and delivered to their desktop electronically. As of July 1998, there were over 7,100 subscribers to the e-mail notification service. According to NASA, user feedback showed that this service was the most popular feature of NAIS. One of NASA's goals in establishing NAIS was to streamline and standardize the agency's procurement process by using the Internet as the gateway to contracting opportunities for businesses. NAIS streamlined or eliminated many of the steps required by the paper-based process for publicizing synopses of contracting opportunities and issuing solicitations. The NAIS process involves fewer steps for procurement staff than the paper process. The EPS, for example, automatically formats notices, performs edit checks, automatically posts notices on the Internet through the NAIS server, and transmits them to the Government Printing Office for publication in the CBD. Using the paper process, procurement staff have to perform analogous functions manually and must also respond to requests for copies of solicitations by preparing mailing labels and mailing the copies. With NAIS, the solicitation files are available for review and retrieval ("downloading") through the Internet. We believe the NAIS process is also easier for contractors to use. Using the paper-based system, contractors must either scan the CBD or periodically call or visit each of NASA's procurement offices to identify contracting opportunities. In contrast, using NAIS, contractors can search for and view as many synopses and solicitations as they wish on-line. They can search by specific commodity, service, or location; view and provide comments on draft solicitations; and access the Federal Acquisition Regulation, NASA procurement policies and procedures, and NASA procurement forms. NASA officials point to the absence of protests from contractors concerning NAIS and the lack of complaints about NAIS to the NASA Ombudsman Program as evidence that the contractor community approves of NAIS. Both businesses and NASA procurement staff have provided positive feedback about NAIS. Overall, industry responses to specific NAIS operational features were positive. In a tally of on-line comments from NAIS users, 87 percent of respondents said they found NAIS to be beneficial. The respondents identified themselves primarily as small, small disadvantaged, or women-owned businesses. For the most part, feedback from the procurement staff was positive, and comments were often similar to those made by businesses. Federal electronic commerce officials from other agencies and industry representatives told us that systems with the operational features of NAIS can benefit both industry and government, especially when all buying organizations are using the same system. An on-line NAIS feedback system was established in 1995 to obtain input, especially from businesses, about the system. The NAIS team also requests feedback from the procurement staff at NASA centers. This feedback was compiled in December 1997 and again in June 1998. The team compiles survey feedback data, develops a consolidated summary of industry suggestions, and uses the summary to determine which changes and enhancements should be made. NASA's evaluation plan for the Midrange Pilot Program did not include a separate NAIS evaluation plan. In May 1998, NASA published the NAIS Business Case, which provides details of current NAIS features, along with its history, benefits, and funding alternatives. However, NASA did not collect data on or evaluate individual procurements advertised through NAIS. The information needed to determine how much time NAIS actually saved or to quantify other NAIS benefits, such as cost avoidance for the procurement process, was not available. We did not find any evidence that NAIS had negative effects, and there is substantial qualitative evidence of the positive impact of NAIS. Appendix II provides more information on our assessment of the NAIS Business Case. Although FASA allowed NASA a 4-year test authority to waive CBD publication requirements and waiting periods up to a limit of $100 million in life-cycle costs, NASA did not establish a mechanism to track compliance with the waiver provision. NASA officials told us that when the waiver authority was originally requested, they anticipated that NAIS, though not yet developed, would be used primarily by the Marshall Space Flight Center, the focal point of the pilot test, and that the $100-million ceiling would be adequate to cover Marshall's procurements. However, the Internet posting capability was rolled out to all NASA centers by April 1995 and was being used agencywide for all competitive procurements over $25,000 1 month before NASA began waiving the CBD notice requirements in October 1995. In early 1997, NASA officials realized that the $100-million ceiling authorized by FASA for the test would not cover agencywide use of NAIS. In May 1997, the Administration submitted a legislative proposal to increase the ceiling for NASA's use of the waiver authority and extend the test period from 4 to 6 years. Congress did not act on the proposal. NASA used the CBD waiver authority through the end of the fiscal year-September 30, 1997. NASA calculated that it awarded approximately 1,585 contracts worth a total of about $177 million using the CBD waiver authority. In an effort to determine the impact of NAIS on the amount of competition for awards, NASA compared the number of vendors' proposals received before and after NAIS implementation. NASA compared the number of offers received per contract action in fiscal year 1994-the year prior to agencywide use of NAIS-to the number of offers per contract action in fiscal year 1997-a year in which all midrange procurements were only advertised through NAIS. As table 1 shows, the average number of offers received per acquisition increased from 6.1 to 7.2 after NAIS implementation. NASA said in the NAIS Business Case that the increase in the average number of offers received per acquisition was especially promising because the majority of these acquisitions were set aside for small businesses. We also calculated and compared the percentages of contract actions and dollars awarded to small businesses in fiscal years 1994 and 1997, respectively. Table 1 shows these calculations. The percentage share of awards to small businesses did not change significantly after NAIS implementation, but the percentage share of dollars awarded to small businesses increased substantially in 1997. A NASA official said this was due in large part to a drop in awards to large businesses and not to any substantial change in dollars awarded to small businesses. Today most federal agencies use the Internet in some way to advertise their contracting opportunities. Vendors search all these various Internet sites for government procurement information. Recent legislation requires the government to provide direct access to notices of agencies' requirements and solicitations through a single governmentwide electronic point of entry. NASA, GSA, and other agencies are testing a multiagency electronic posting system. Agency officials are concerned that the benefits of providing direct electronic access to contracting opportunities may be offset by existing statutory requirements for publication of procurement notices and minimum waiting periods. According to the San Antonio Electronic Commerce Resource Center,more than 400 different Internet sites provide federal procurement information. This multitude of sites makes it difficult and time-consuming for contractors to obtain governmentwide information on contracting opportunities. The March 1998 report, Electronic Commerce for Buyers and Sellers, A Strategic Plan for Electronic Federal Purchasing and Payment, described planned enhancements to the electronic version of the CBD, CBDNet, which now provides a single point of entry only for procurement notices. GSA and NASA approached the Department of Commerce and the Government Printing Office to form an interagency team to implement an enhanced CBDNet and to make an Internet-based EPS with features comparable to those of the NAIS system available to the entire federal sector. However, the agencies could not reach agreement on an enhanced CBDNet implementation approach. Among unresolved issues were the source and level of funding needed for the project, the costs to develop and implement the system, and the potential financial impact on the operating and user agencies. NASA, GSA, and other members of the Interagency Acquisition Internet Council are developing and testing a pilot multiagency posting system derived from the NAIS EPS to enable agencies to post notices, solicitations, and other acquisition-related documentation directly onto the Internet, where it would be available through a single point of entry. GSA is leading this effort and spent over $400,000 in the past year developing this multiagency EPS software. In October 1998, GSA began using the pilot software as the agency's single EPS. If testing demonstrates that the multiagency EPS is capable of providing effective access to notices and solicitations through a single point of entry, the Administrator of the Office of Federal Procurement Policy will consider designating it as the single governmentwide point of entry required by section 850. The difficulty of establishing a single governmentwide electronic point of entry for federal procurement opportunities was emphasized by several agency officials, who said they were concerned about the costs involved in developing, operating, and maintaining such a system; cost sharing among different agencies; and the linkages that need to be established with existing agency systems through the Internet. They also questioned how and when decisions would be made concerning who would lead such a project, who would be responsible for operating and maintaining the system, and who would mandate its use by agencies. Agency officials also said that the benefits of a governmentwide electronic notice capability could be enhanced through changes to the current statutory minimum waiting periods for notices and solicitations, which were fashioned around a paper-based process. Acquisition officials noted, in particular, that the law requires a 15-day waiting period after publication of a notice before a solicitation can be issued, even though the Internet enables agencies to publicize notices and provide solicitation files simultaneously, giving contractors immediate access to these documents and saving time in the procurement process. Agency officials said that changing these requirements could increase the benefits of using electronic commerce. OFPP plans to study what type of legislative action may be needed to enhance the benefits of electronic publication of business opportunities. NASA has shown that it can successfully use NAIS to effectively disseminate solicitations and other procurement information, giving contractors easy and immediate access to business opportunities at each of its geographically dispersed procurement offices. Feedback from users has been positive, and the system has allowed increased standardization of the procurement process agencywide. However, benefits such as the time savings obtained through use of the system cannot be readily quantified because of data limitations. While NASA believes that NAIS has had a positive impact on competition, this conclusion was based on only 2 years of data. NASA's on-line electronic posting system was the model for the pilot multiagency EPS software that is being tested by GSA, NASA, and several other agencies. If successful, this EPS may be considered for governmentwide expansion and designation as the single governmentwide point of entry for both notices and solicitations. While the March 1998 report assessing electronic commerce initiatives discussed enhancements to the CBDNet, it did not discuss the multiagency EPS pilot program. If a governmentwide EPS is successfully implemented, its benefits potentially could be enhanced by changing the statutory notice and waiting periods. We recommend that the Administrator of the Office of Federal Procurement Policy describe the current plan for implementation of a governmentwide single point of entry to procurement information in its next annual report to Congress. This plan should take into consideration the status of and lessons learned from the multiagency EPS pilot and its potential as the basis for a comprehensive governmentwide electronic posting system. We also recommend that the Administrator of the Office of Federal Procurement Policy determine whether the issues raised by agency officials concerning procurement notice requirements set forth in statute, including waiting periods, require legislative action and make recommendations to Congress on legislative changes, if necessary. In commenting on the draft of this report, NASA and OFPP generally agreed with our findings and recommendations. NASA stated that it was proud of the NAIS staff for pioneering the use of the Internet in federal procurement; and our review largely corroborated the agency's assessment of the value of NAIS. OFPP stated that NAIS has created the foundation for the ongoing multiagency pilot of an electronic posting system. If successful, the EPS will provide easy access to procurement information across the government through a single point of entry. OFPP was encouraged by the promising findings identified in the report. OFPP also commented on our recognition that current statutory requirements for procurement notices and minimum waiting periods may limit the potential benefits of electronic commerce and indicated that the feasibility of legislative action would be studied. The comments from NASA and OFPP are reprinted in their entirety in appendix III and IV, respectively. To determine whether the NAIS electronic notice and publication system was an effective mechanism for disseminating procurement information, we obtained information about the system's design, development, operation, costs, and benefits. We obtained documents from and interviewed officials at NASA Headquarters in Washington, D.C., and at the Marshall Space Flight Center (the lead center for NAIS project management) in Huntsville, Alabama. We also accessed NASA's procurement Internet site (http://procurement.nasa.gov) and reviewed the on-line NAIS operational features available to both public users and NASA contracting staff to determine whether the system worked as NASA represented. To evaluate NASA's claim that NAIS usage had a positive impact on competition and was beneficial to small businesses, we compared NASA procurement data for fiscal year 1994 (the year prior to agencywide use of NAIS) with data for fiscal year 1997. We did not independently verify the NASA procurement data obtained from the NASA acquisition management information system. We also reviewed NASA's evaluation of the system contained in the May 1998 NAIS Business Case. We reviewed the data in the NAIS Business Case, including calculations of cost avoidance and other benefits attributed to NAIS. We examined on-line industry feedback and narrative responses from vendors and NASA buyers about NAIS. To determine the agency's compliance with statutory requirements, we reviewed the goals, objectives, and time frames established by FASA for NASA's test of electronic notice and publication. To assess the status of efforts to establish a governmentwide EPS based on the NAIS model, we asked electronic commerce coordinators from several federal agencies, federal Electronic Commerce Resource Centers, and two industry groups for their observations. We received responses from coordinators at five agencies, two resource centers, and one industry group. We also interviewed acquisition officials at GSA, the Department of Commerce, and OFPP to obtain information on interagency efforts to develop a governmentwide electronic procurement posting system. We also reviewed the goals, objectives, and milestones established for the governmentwide electronic commerce program by the National Defense Authorization Act for Fiscal Year 1998. We performed our work between January and December 1998 in accordance with generally accepted government auditing standards. We are sending copies of this report to the Administrator, NASA; the Administrator, OFPP; and the Director, Office of Management and Budget. We will also make copies available to others upon request. Please contact me at (202) 512-4841 if you or your staff have any questions concerning this report. Major contributors to this report are listed in appendix V. Staff at the National Aeronautics and Space Administration (NASA) responsible for developing and implementing the NASA Acquisition Internet Service (NAIS) consisted of a small group of procurement and, in some cases, technical staff from each NASA center and headquarters. They adopted an incremental approach to system development and operations. This approach included the following steps: (1) general requirements were drawn up, (2) commercially available software options were researched, (3) a prototype was designed and developed if a commercial solution was not available, (4) a prototype for testing was installed, (5) requirements were refined and a small pilot initiated, and (6) the pilot was expanded and the software solution added to the NAIS system. NASA's development of its Electronic Posting System (EPS) is characteristic of the way the NAIS team applied the concept of incremental design, development, and implementation of NAIS features. Prior to agencywide use of EPS, each NASA center had its own unique process for preparing and posting procurement notices and solicitations on the Internet. Some centers used word processors while others used some form of automated document generation system. The completed files were then provided to a Web curator at the center who would post them on an Internet server and manually update a readable index of files. Changes to the posting process occurred spontaneously at various centers with little central control or agreement. The 10 centers' differing processes required 10 separate efforts to implement a single change that had to be agreed on by the larger NAIS team. These process differences also made it difficult to maintain a standard "look and feel" when searching out business opportunities. A vendor was faced with 10 different entry points to search. NAIS operated for a year in this fashion. Then an ad-hoc group of NAIS team members began to study ways to make the centers' posting processes more uniform. The team concluded that the NAIS posting process should be centralized to a single server and that the application for posting and maintaining documents should be standardized by using on-line forms to be completed by users. NAIS technical team members then designed and developed the uniform Web posting tools that were first piloted and then deployed agencywide. According to NASA, implementation of the EPS eliminated much duplication among the centers, standardized posting practices across all NASA centers, and lowered NAIS maintenance costs. The EPS application provides a common look and feel of acquisition opportunities. The incremental, evolutionary approach to deploying an electronic posting system avoided the costs and delays often associated with attempting a single solution to standardize disparate, decentralized systems. Additionally, because much of the system is based on commercial standards, commercially available software, and standard hardware, it was easily adopted by each of the centers without major disruptions. All NAIS features have been built using commercial Internet standards and Web methodology so that even casual Internet users can easily use NAIS. For internal procurement staff using standard NASA computer hardware and software, NAIS tools may include user instructions. For example, the NAIS EPS has an on-line user's manual linked directly to the application. For external users with a wide variety of hardware and software, NAIS features are based on widely accepted Internet protocols and standards, and the features are designed to be used without any formal training. In May 1998, NASA completed and published an evaluation of NAIS in a document called the NAIS Business Case. The document sought to justify continued funding for NAIS by showing that the system's quantitative and qualitative benefits far exceed NASA's investment. According to NASA's assessment, the NAIS tools implemented to date had produced outstanding results. We found weaknesses in the evaluation's methodology. For example, NASA claimed that the Midrange Pilot initiatives, including NAIS, reduced procurement leadtimes by 40 percent. We examined the data used by NASA to support this calculation. In a number of cases, the data showed that the official purchase requests arrived at NASA procurement offices after solicitations for the requested products or services had already been issued. Consequently, we could not verify NASA's claimed 40-percent reduction in leadtime. However, it is obvious that the reduction in mail delays and in the manual steps and efforts needed to post a procurement notice should contribute to reduced leadtimes. We also found that the NAIS Business Case's discussion of the resources used in developing and operating NAIS was incomplete. It described only the full-time equivalent (FTE) staff years of effort, both funded and unfunded, which were devoted to NAIS development and support in fiscal year 1997 and did not reflect the additional resources such as technical support, hardware, software, and communications services provided by various NASA centers. Because the evaluation only identified NAIS FTE staffing for 1 year and did not identify hardware, software, and other costs associated with NAIS, we asked NASA to provide fiscal year estimates of the total costs associated with NAIS development and ongoing operations. The table below depicts NASA's estimates of the staffing, hardware, software, and miscellaneous costs associated with NAIS for fiscal years 1995 through 1998. Amounts for fiscal years 1995 through 1997 are estimates, while the amount for fiscal year 1998 is a projection. Hardware/ software/misc. NASA reported in the NAIS Business Case that data from the NAIS on-line feedback system showed that over 85 percent of those who responded to the question, "Is NAIS, as a whole, beneficial?," answered "yes." We reviewed summary reports of responses tallied from the NAIS system and found that of the 2,310 persons answering this question, 2,007 answered yes (87 percent). The survey respondents identified themselves primarily as small, small disadvantaged, or women-owned businesses. Overall, we found both industry and NASA procurement staff comments about specific NAIS operational features were positive. We also reviewed feedback responses submitted from 1995 through June 1998 to an open-ended question about enhancements that would improve NAIS. As of June 30, 1998, the NAIS team had completed actions or had actions underway related to 42 of 89 suggestions. Federal officials and groups representing industry perspectives told us that the NAIS Business Case properly identified the areas in which both government and industry may experience time and cost savings through an effective agencywide EPS. They noted that electronic commerce systems with the functionality and user-friendly features of NAIS have clear potential to benefit both industry and government, particularly if all buying centers are using the same system. John Carter The first copy of each GAO report and testimony is free. Additional copies are $2 each. Orders should be sent to the following address, accompanied by a check or money order made out to the Superintendent of Documents, when necessary. VISA and MasterCard credit cards are accepted, also. Orders for 100 or more copies to be mailed to a single address are discounted 25 percent. U.S. General Accounting Office P.O. Box 37050 Washington, DC 20013 Room 1100 700 4th St. NW (corner of 4th and G Sts. NW) U.S. General Accounting Office Washington, DC Orders may also be placed by calling (202) 512-6000 or by using fax number (202) 512-6061, or TDD (202) 512-2537. Each day, GAO issues a list of newly available reports and testimony. To receive facsimile copies of the daily list or any list from the past 30 days, please call (202) 512-6000 using a touchtone phone. A recorded menu will provide information on how to obtain these lists.

Summary:
Pursuant to a legislative requirement, GAO reviewed the National Aeronautics and Space Administration's (NASA) Acquisition Internet Service (NAIS), focusing on: (1) whether NAIS is an effective mechanism for disseminating procurement information to industry, including small businesses; and (2) the status of efforts to develop a governmentwide electronic procurement information system similar to NAIS. GAO noted that: (1) NAIS is a simple, effective, and user-friendly system for disseminating information on contract opportunities; (2) NAIS has contributed to the development of a more standardized and streamlined acquisition process at NASA and provides a central electronic source of procurement information from NASA's decentralized facilities; (3) it allows businesses to obtain procurement information immediately, without waiting for mail delivery of printed information; (4) vendors especially like the electronic mail notification service that automatically sends announcements about procurements of interest to them; (5) vendor feedback about NAIS came primarily from small businesses and was generally positive; (6) procurement data showed that offers and awards to small businesses did not change significantly after NAIS implementation; (7) NASA noted that data limitations made it difficult to quantify other NAIS benefits; (8) NASA, the General Services Administration, and other federal agencies are working together to develop a single, governmentwide Internet entry point for information on federal procurement opportunities; (9) but a number of steps must still be taken and many obstacles remain; (10) even if the new system is successfully developed and implemented governmentwide, current statutory requirements for publication of procurement notices and minimum waiting periods for mail delivery may continue to limit the potential benefits of an electronic procurement information system; and (11) the same legislation that encouraged NASA and others to work together also requires the Office of Federal Procurement Policy to submit to Congress annual reports assessing compliance with the requirement to provide direct access to procurement information through a single governmentwide electronic point of entry.