DOD has long been concerned about the quality of service it provides military personnel and their families when they relocate. DOD reported that in 1996 it paid roughly $2.8 billion to move 800,000 families. Despite the fact that it moves more household effects than any U.S. corporation, DOD has found that its system has provided military personnel some of the worst service in the Nation. DOD has reported that 25 percent of its moves have resulted in damage claims, compared to 10 percent of the moves undertaken for the private sector. Further, best-in-class movers have had customer satisfaction rates of 75 percent, compared with 23 percent for DOD personnel moves. The Army recognizes that in today’s environment of limited resources, it cannot continue to move personal property in the usual way. Currently, the personal property program is run centrally by the headquarters office of the Military Traffic Management Command (MTMC) and relies almost exclusively on more than 1,250 motor van carriers and forwarders to handle its domestic and international moving needs. The current system is not specifically designed to select carriers on the basis of quality service; rather, carriers offering a minimally acceptable level of quality are generally selected based on the lowest rates. DOD offers carriers meeting the low rate on each route the opportunity to share in any traffic moving on that route. Further burdening the current system are “paper companies,” which are affiliates of larger or parent companies established to increase shares in DOD traffic. These companies increase DOD’s administrative activities but do not add hauling and storage capacity. To test whether commercial business practices can help alleviate these problems and to acquire quality service for its military personnel and families during the relocation process, the Army began developing a pilot project in February 1996 to outsource transportation and relocation services. The company selected for outsourcing was expected to offer Army personnel and their families a wide range of relocation services. In addition to packing, transporting, and storing household goods, the company was to provide other services that included entitlement counseling; point-to-point move management; help with selling, buying, or renting a home; and property management and mortgage services. On January 31, 1997, the Army selected PHH Relocation, a relocation/move management company, now named Cendant Mobility, of Bethesda, Maryland, from among 11 offerors as the contractor for its pilot project. The evaluated price for 3 years—a base year and two 1-year option periods—was $22.5 million. The pilot program involves Army personnel stationed at Hunter Army Airfield moving household goods, unaccompanied baggage, mobile homes, and boats to other authorized locations, including Georgia, other states in the continental United States, Alaska, Hawaii, and foreign countries. It excludes privately owned vehicle shipments and shipments moving into nontemporary storage. Prior to the pilot project, Hunter was part of the MTMC-managed program at Fort Stewart, Georgia, although Hunter staff arranged the moves. Contract operations at Hunter Army Airfield began on July 1, 1997. The contractor, Cendant Mobility, receives the authorizations for relocation from Army members and arranges to move household goods, unaccompanied baggage, and mobile homes from Hunter to worldwide destinations. It also assists a member who may want to make a Do-It-Yourself personal property move. Data provided by the Army on the first 6 months of contractor operations indicated that Cendant had received and processed 793 relocation authorizations and arranged (booked) 581 moves. These moves included 244 domestic (within the continental United States) household goods shipments, 227 international household goods shipments, 106 international unaccompanied baggage shipments, and 4 mobile home shipments. Cendant also assisted members in initiating 181 Do-It-Yourself moves. (See table 1.) The Army has developed an evaluation plan to facilitate data collection on quality of life, total cost, and impact on small business; define agency responsibilities; and explain which data will be used for comparative purposes. It has indicated that data will be collected for a minimum of 12 months. Quality of life is defined primarily in terms of customer satisfaction, measured principally from customer surveys at both Hunter and Fort Stewart. Additionally, quality of life will be assessed in terms of claims for loss and damage based on incidence, amounts, and settlement times. Total costs include transportation, storage, administration, and overhead. Transportation and storage costs under the pilot will be compared with what the Army would have paid under the MTMC program. Administrative and overhead costs for the pilot will be compared with the cost of doing business under the MTMC program. Claims costs under the pilot will be compared with claims costs on shipments made in calendar year 1995. Impact on small business will be based on the percentage of moves, tonnage, and dollar amount awarded to small and disadvantaged businesses under the pilot compared with calendar year 1995 experience. The Army Audit Agency has been tasked to validate all the data used to conduct the Army’s evaluation. Prior to commenting on a draft of this report, the Army had not explained what constituted success for the pilot, how much weight each factor (quality of life, total cost, and impact on small business) should have in determining overall success, and whether there were thresholds that specific factors should attain to be considered successful. Such advance determinations are typically part of a sound study methodology to enhance the credibility of the results and to avoid any perception of bias. In response to our draft report, DOD provided us with additional information it had received from the Army. The information provides thresholds for determining the success of the overall Hunter pilot and the relative importance of each evaluation factor. While the Army has not finalized its evaluation method, the new information clarifies the description of the three evaluation factors, establishes a point-scoring system and threshold for each factor and subfactor, and prescribes the relative weight of each factor for determining the pilot success. We will continue to evaluate the Army’s evaluation method as it is finalized and have included the new information in the following sections of this report. We recognize that the relationship among the factors—quality of life, costs, and impact on small business—complicates the process of setting absolute targets for each factor or for assessing the trade-offs among the factors. However, the Army may consider certain levels critical for certain factors (e.g., the budget may limit the amount of increased costs that can be paid for increased quality of life levels, measured by customer satisfaction survey and claims experience). On the other hand, there may also be certain customer satisfaction levels (e.g., those similar to the commercial market) that the Army expects to obtain. Therefore, making these kinds of decisions early in the evaluation process should provide the Army a more definitive basis for determining whether the pilot project is successful. Accordingly, the Army has taken action in response to our draft report recommendation that it determine what results it requires to judge its pilot a success. The Army’s evaluation plan indicates that overall, quality of life is the most important of the evaluation factors. The additional information recently provided suggests that the Army will use a point-scoring system to determine the relative weight of each factor and subfactor, as shown in table 2. If the scores of the pilot total 3.75 or more, the Army will consider the pilot a success; if below 3.75, or if any of the individual factor scores falls below its threshold, the Army will consider it unsuccessful. Quality of life is the key element the Army is considering in assessing the Hunter pilot. The Army’s most recent information indicates that quality of life will be based on results of four subfactors—customer satisfaction, claims settlement time, percentage of direct deliveries, and number of congressional or White House inquiries concerning dissatisfaction of Hunter test participants. Customer satisfaction will be determined by examining responses from two surveys: (1) Cendant will obtain comments made by all pilot test members after their moves and (2) an independent contractor will survey both the pilot test members and those making similar moves at Fort Stewart under the MTMC program. Responses from both surveys will be compared. Although the surveys will be quite detailed, the single question for determining pilot test success will be “ How satisfied are you with your relocation moving experience?” The Cendant survey consists of 12 questions, the key question being: “How satisfied are you with your relocation moving experience?” Responses are used to determine the amount of “performance” pay that the contractor can earn in accordance with the contract terms. Data provided to us for the first 6 months of activity, covering 88 responses where the response choices were “excellent,” “very good,” “good,” “fair,” and “poor,” showed that 98 percent of the members had received “excellent,” or “very good,” service. None indicated dissatisfaction with their overall relocation moving experiences. The Army Audit Agency, however, has not yet completed its validation of those responses. The independent contractor survey is more extensive and probing than the Hunter contractor survey, consisting of about 100 questions. For example, it includes a number of questions on the members’ previous moves and services used. The key question, however, is: “How satisfied are you with your relocation moving experience?” When the Hunter participants are at least 10 percent more satisfied than the control group, which are the personnel moving from Fort Stewart during the same time as the Hunter personnel, the customer satisfaction rating subfactor will be deemed successful. While there are a number of possible responses to this question (“very satisfied,” “satisfied,” “neither satisfied nor dissatisfied,” “dissatisfied,” or “very dissatisfied”), it is not clear how “more satisfied” will be defined. Results to date are limited because the independent contractor did not start until October 5, 1997, and it needed time to test and revise the survey before proceeding with the full survey process. The survey results are expected to be reported to the Army in October 1998. The second subfactor related to quality of life is claims settlement time. This is based on the contract requirement that the contractor offer settlement to a member within 30 days of claim receipt for damage or lost property. Consequently, the threshold is 30 days. Information on claims data is preliminary and may take some time to fully accumulate since servicemembers may file claims to the Army up to 2 years from delivery of property. As of December 31, 1997, however, over 500 shipments had been arranged by the contractor, and 33 of the 40 claims filed by customers against the contractor had been closed. The average time it took the contractor to settle claims was 13 days, the average amount claimed was $919.18, and the average amount settled was $455.15. To date, none of the Hunter servicemembers has exercised his or her option to file a claim with the Army. The third subfactor is direct delivery, which is defined as delivering a shipment to a servicemember’s residence without storage intransit. Direct delivery is a contract performance measure in which the contractor is paid an incentive for maintaining a direct delivery rate above 60 percent. Consequently, the threshold is 60 percent. Of the shipments delivered during the first 6 months of contractor operations, 35 percent were direct deliveries. The fourth subfactor under quality of life will be the number of congressional or White House inquiries that the Army Office of Deputy Chief of Staff for Logistics will receive concerning dissatisfaction of a Hunter test participant. The established threshold is one inquiry. Total cost is another factor that will be used in the pilot evaluation, and the carrier industry has expressed concerns that the pilot program could cost the Army more than it would incur under the MTMC program. According to the Army, total cost is the sum of (1) transportation, which includes line-haul transportation, accessorials, and storage prior to delivery; (2) claims, those costs paid by the Army; and (3) management cost, which includes pay for performance, management price, and overhead and administrative expenses. The Army has indicated that the Hunter pilot will be determined successful if its total cost does not exceed the baseline cost, which is the cost, including management cost, that the Army would have incurred had the shipments moved at MTMC program rates, by more than 30 percent. Reported pilot costs are still preliminary. To date, the only validated pilot cost data relate to transportation, storage, accessorial services, claims, pay for performance, and management fees. Through late January 1998, the Army paid the contractor $503,835 for 203 moves—150 full service shipments and 53 Do-It-Yourself moves (see table 3). The costs for these same shipments and moves, priced at the current MTMC program rates, are being developed but are still being validated. Developing overhead costs has historically been difficult in government agencies, including DOD, because such data are often unreliable and unavailable. The Army, however, is collecting data on administrative or overhead costs, which include costs for personnel, supplies, claims processing, voucher processing, inbound shipment processing, electronic automation, excess cost processing, building overhead, telephone and copier, and MTMC overhead. Even though overhead and administrative costs for both the pilot and the comparable MTMC program are being developed, they have not yet been fully validated by the Army Audit Agency. The Agency’s responsibilities include providing the Army program managers with guidance in capturing administrative and overhead costs, guidance in statistical sampling techniques, and the validation of the test and baseline data. The impact of the Army’s pilot on small businesses has been a source of concern to the carrier industry. The Army is collecting data on the percentage of moves, tonnage, and dollar amounts awarded to small and disadvantaged businesses, including carriers, forwarders, and their agents, under the pilot program. It will then compare that data with calendar year 1995 Hunter data. The Hunter contractor is a relocation firm who subcontracts or who has written agreements with various service providers. Some of these providers are carriers and some are carriers’ agents. The Army has indicated that it will measure impact on both types of providers and has established a threshold of 23 percent to consider the pilot small business participation a success. It is not clear, however, which small business groups—carriers, agents, or both—on which measurements will be taken to establish small business participation in the pilot. As of December 31, 1997, 74 of 533, or 14 percent, of the shipments booked by Cendant during the first 6 months were awarded to small businesses.In comparison, 32 percent of the calendar year 1995 shipments booked by Hunter under the MTMC program were awarded to small businesses. The data are still being verified by the Army Audit Agency. According to Cendant, moving industry opposition to DOD’s reengineering plans and participation of relocation companies, such as Cendant, in DOD and civilian government household goods business has had a major impact on the availability to Cendant of small business subcontractors in the Hunter pilot. According to Cendant, a number of major van lines have been reluctant to sign contracts with it for the pilot. Eight of the 15 small business agents who had been doing business with Hunter are affiliated with van lines that have not signed Cendant contracts. In December 1997, Cendant initiated efforts to obtain signed contracts and broaden the base of potential small business certified agents who could participate in the pilot. Specifically, on December 8, 1997, Cendant held a meeting in which 20 officials, representing 15 local moving companies, expressed an interest in providing relocation services. According to Cendant officials, several of the local carriers will ask to be considered for future business and Cendant intends to continue encouraging small business participation and increase its small business supplier base. Even as efforts are being made to increase small business participation in the Army’s pilot, a long-standing issue remains—the lack of verifiable data establishing company size and ensuring that companies qualify as small businesses. Historically, such data have been based on carrier/forwarder self-certification, which MTMC has considered questionable. Under the MTMC program, company size is established when MTMC initially approves companies’ participation in the program. MTMC, however, does not verify the data or require the data to be updated. In addition, MTMC has informed us that, in its view, some businesses have certified themselves as small, but, in fact, they rely exclusively on the assets of their parent companies to perform the actual transportation services. Although these so-called “paper companies” have certified themselves as small businesses, when the revenues of their parent companies are considered, MTMC believes these companies may exceed the small business size threshold. Moreover, few of the MTMC-approved carriers and forwarders, whether paper companies or not, are identified in the Small Business Administration data files. Consequently, carrier/forwarder size determination will be difficult, and the Army’s ability to compare test data with baseline data could be questionable. Other pilots or tests are also underway or are being planned in DOD. MTMC, as DOD’s overall personal property program manager, is contracting for a pilot covering selected shipments moving from all military bases and installations throughout North Carolina, South Carolina, and Florida. This pilot will use one or more companies to handle moves on specific traffic channels—from one state to a group of states, excluding Hawaii and Alaska, and to various locations in Europe—under long-term contracts. Key goals of this pilot are to simplify DOD’s current transportation acquisition process and to substantially improve and put on par with corporate customer standards, the quality its military personnel and their families receive from DOD’s contracted movers. This pilot differs from the Army Hunter pilot in that it is larger, covering a wider geographical origin area and more shipments, and involves the relocation of members of all branches of service. The Army Hunter pilot, on the other hand, is limited to one location, Hunter, and one service, the Army, although it provides for move management services, including entitlement counseling and destination relocation assistance, for all Hunter Army families, relocating to anywhere in the world. Although the original MTMC solicitation was issued on March 14, 1997, the start of the MTMC pilot has been delayed—first, by a disagreement between DOD and the moving industry over the approach to take for the pilot test and, second, by a series of protests filed with us concerning the terms and conditions of the solicitation. The Navy began a test in January 1998 to use commercial practices at its personal property shipping office at the Fleet Industrial Supply Center-Puget Sound, Bremerton, Washington. The initial test is limited to moves being made to five U.S. destinations. It allows sailors to select from a list of eligible moving companies the company they would like to move their property. Like the ongoing Hunter and proposed MTMC pilots, sailors receive counseling, full replacement value protection for any loss or damage, and direct claims settlement with the contractor or moving company. The U.S. Transportation Command (USTRANSCOM) plans to evaluate the Hunter pilot and the two other military personal property pilots or tests. USTRANSCOM will also maintain oversight of the personal property program pilots and tests and assure consistent evaluation criteria and assessment of the pilot program results. Upon completion of the pilots and tests, and in coordination with each service, USTRANSCOM will determine which pilot or test, or portions thereof, if any, could provide better long-term results for DOD. It will then recommend the follow-on course of action and time-lines for implementation throughout DOD, if applicable. We have previously provided informal comments to USTRANSCOM on their evaluation plan and will continue to monitor the personal property pilot tests. Notwithstanding this and other pilots and tests planned or underway, DOD and Army officials briefed us at the end of this review about plans to expand the Hunter pilot by testing it at selected DOD sites. Specifically, the expansion would include a higher shipment volume (between 40,000 and 50,000 shipments per year) and a larger range of services (such as adding nontemporary and vehicle storage) to determine if the Hunter pilot results can be successfully duplicated throughout DOD. DOD would like to begin implementing the test by October 1, 1998. However, to date, all of the expansion details have not been worked through, such as determining how the expansion will be managed and evaluated and identifying all of the installations and services that would participate. The Army is well underway in implementing its pilot project for outsourcing relocation services at Hunter Army Airfield. It has identified quality of life, costs, and impact on small businesses as important factors to be used to determine the success of the pilot. In response to our draft report recommendation, the Army provided additional information that states what results it will use to judge the pilot a success. The information, with some minor exceptions, represents important steps toward formulating a sound evaluation plan. However, the evaluation method has not been finalized or made a formal part of the evaluation plan, and questions remain about the definition of terms and small business group measurements. We recognize that further issues could be identified as the evaluation method is finalized. In addition, the Hunter pilot is only one initiative among several that will be tested to determine how DOD can improve the delivery of relocation services to military personnel. Should DOD decide to expand the Hunter pilot to include significantly more military customers and services, this could provide additional information for assessing the concept and for evaluating results and savings achieved. However, USTRANSCOM, which has primary responsibility for analyzing overall results and savings achieved, will ultimately need to assess the relative strengths and/or limitations of each pilot project to determine how DOD can best improve the personal property program. We recommend that the Secretary of Defense direct the Secretary of the Army to finalize its method for evaluating the pilot results to include the new evaluation information provided in response to our draft report. In doing so, the Secretary of Defense should direct the Secretary of the Army to strengthen the evaluation plan by better defining terms and more clearly defining the small business groups on which measurements will be taken to establish small business participation in the pilot. DOD stated that it concurred, with comment, with the report and recommendation. Our draft report contained a recommendation that the Secretary of Defense direct the Secretary of the Army to determine what results the Army will require to judge its pilot a success, such as specifying the relative weight of each factor it is measuring and determine if certain factors must meet certain thresholds, and if so, specify those factors and levels. In agreeing with our draft recommendation, DOD provided additional information regarding the Army’s method for determining what results it plans to use to judge the pilot a success. While we must reserve conclusions until an evaluation method is finalized, our initial assessment of the additional information provided indicates that it begins to clarify the description of the three evaluation factors (quality of life, total cost, and impact on small business), establishes a point-scoring system and threshold for each factor and subfactor, and prescribes the relative weight of each factor for determining the pilot’s success. To the extent this information is incorporated into the Army’s evaluation plan, it will provide a more complete basis for the Army’s analysis. However, our initial assessment of the information indicates that the Army could strengthen the evaluation by better defining terms, such as “more satisfied” in reference to comparing customer satisfaction ratings and by more clearly defining the small business groups—carriers, agents, or both—on which measurements will be taken to establish small business participation in the pilot. Consequently, we have revised our report to discuss the additional information and modified our recommendation to indicate that the evaluation method should be finalized and be made a formal part of the Army’s evaluation plan. DOD also had comments on three specific areas. First, DOD disagreed with our references that the Hunter pilot was being expanded “DOD-wide.” DOD indicated that while expansion of the Hunter pilot is being considered as part of its total reengineering effort, it has not approved or decided to expand the program “DOD-wide.” We revised the report to reflect the current status of DOD’s plans. Second, DOD stated that characterizing the DOD overhead cost data as often unreliable and unavailable undermines the entire effort. We acknowledge DOD’s position; however, the Army Audit Agency’s validation of overhead data will not be completed for several months. Further, while the preliminary data represent a full 6 months of performance, the Army has not developed the same full 6 months of transportation cost data. (The data are needed to accurately portray the pilot’s cost.) Until the Army has developed the cost data both for overhead and transportation for the full 6 months, and the audit validation process is complete, any conclusions about the total cost of the pilot are premature. Third, DOD expressed concern that we had not mentioned that greater small business participation in the Hunter pilot could be realized if the parent companies of small businesses that previously serviced Hunter agreed to allow their small business concerns to sign contracts with the Hunter contractor. We have added that information in our discussion of the pilot’s impact on small business. DOD’s comments, along with those prepared by the Army, are included as appendix I. We asked the Army for data and supporting documents concerning the first 6 months of activity at Hunter, and the Army provided us that data and documents on February 2, 1998. We reviewed the data and observed how it was being collected. We did not independently verify the accuracy of the data; however, the Army Audit Agency will be validating the data. To date, sufficient data have not been collected for comparison purposes. We also met and discussed matters related to the pilot with officials of the following: Office of Deputy Under Secretary of Defense (Logistics-Transportation Policy), Washington, D.C.; USTRANSCOM, Scott Air Force Base, Illinois; Office of Army Deputy Chief of Staff for Logistics, Washington, D.C.; Hunter Army Airfield, Georgia; Army Audit Agency, Hunter Army Airfield, Georgia; Fort Stewart, Georgia; and MTMC, Falls Church, Virginia. We reviewed the Army’s plans for evaluating the pilot project and reviewed the chronology of the pilot from announcement of plan, presolicitation, solicitation, award, protest, and implementation. Regarding the management of the pilot project, we visited both Hunter Airfield and the personal property shipping office at Fort Stewart and discussed the information being collected by Army officials. We also reviewed the data prepared by the Army and the Army Audit Agency. We met with Hunter contractor officials; observed their contract operations, including the actions of their contracted carrier and agent activities; and observed a relocation counseling session with a Hunter soldier. We observed similar activities at Fort Stewart and were briefed or received documentation on the other planned or ongoing DOD personal property tests. We also discussed matters related to the Army Hunter pilot with three agents in Savannah who are on the MTMC-approved list of agents serving the military in the Fort Stewart/Savannah area. Our review was conducted between July 1997 and March 1998 in accordance with generally accepted government auditing standards. We are sending copies of this report to the Secretaries of Defense and the Army; the Commander in Chief, USTRANSCOM; the Department of Army Deputy Chief of Staff for Logistics; and the Commander, MTMC. We will also make copies available to others upon request. Please contact me on (202) 512-8412 if you or your staff have questions concerning this report. Major contributors to this report are listed in appendix II. The following are GAO’s comments on the Department of Defense’s letter dated May 21, 1998. 1. We have included Tab C containing new information that the Army will use to evaluate the pilot. However, we have not included Tab A and Tab B, which are administrative and technical comments and were addressed in the report as appropriate. 2. An Army Hunter pilot program official advised us that the wording in paragraph 1e (Total Cost) was incorrect and should have indicated that the Hunter pilot will be determined successful if the total contract-case cost of the pilot does not exceed the baseline-case cost by more than 30 percent. Barry W. Holman, Associate Director Nomi R. Taslitt, Assistant Director J. Kenneth Brubaker, Evaluator-in-Charge Barbara L. Wooten, Evaluator Leo G. Clarke, III, Evaluator The first copy of each GAO report and testimony is free. Additional copies are $2 each. Orders should be sent to the following address, accompanied by a check or money order made out to the Superintendent of Documents, when necessary. VISA and MasterCard credit cards are accepted, also. Orders for 100 or more copies to be mailed to a single address are discounted 25 percent. U.S. General Accounting Office P.O. Box 37050 Washington, DC 20013 Room 1100 700 4th St. NW (corner of 4th and G Sts. NW) U.S. General Accounting Office Washington, DC Orders may also be placed by calling (202) 512-6000 or by using fax number (202) 512-6061, or TDD (202) 512-2537. Each day, GAO issues a list of newly available reports and testimony. To receive facsimile copies of the daily list or any list from the past 30 days, please call (202) 512-6000 using a touchtone phone. A recorded menu will provide information on how to obtain these lists.

Summary: Pursuant to a legislative requirement, GAO provided information on the Army's Hunter pilot program, an alternative approach for providing relocation services for its personnel stationed at Hunter Army Airfield, Savannah, Georgia, focusing on: (1) data collected to date; (2) GAO's observations on how the Army plans to evaluate the data; and (3) other pilot tests that are under way or planned in the Department of Defense (DOD). GAO noted that: (1) through the first 6 months of operation, the Hunter pilot contractor had received and processed nearly 800 requests for transportation and relocation services, arranged for transportation of nearly 600 personal property shipments, and assisted Army members initiate another 200 do-it-yourself moves; (2) because many of the shipments had not been completed at the time of GAO's review, the Army had paid for only about 200 of these shipments, at a cost of about $500,000, but it was actively collecting data on all the factors--quality of life, cost, and impact on small businesses--it was planning to use in its evaluation of the pilot; (3) however, at the time GAO completed its work, the Army had not determined what it would consider successful within each factor or how much weight each factor would have in determining overall success; (4) specifically, the Army had not determined criteria to judge the success or failure of the pilot to help it assess whether the pilot was performing better or worse than the existing program; (5) making such determinations before pilot program data are analyzed would be important to enhancing the credibility of the Army's assessment and for use in making comparisons with other pilot programs that are under way or planned in DOD; (6) these include pilots under way by the Navy or planned by the Military Traffic Management Command; (7) also, DOD and the Army are considering expanding the pilot now being tested at Hunter Army Airfield to other DOD sites; (8) subsequently, in response to GAO's draft report, the Army provided it with new information on how it will determine the results of the pilot; (9) the information, with some minor exceptions, represents important steps toward formulating a sound evaluation plan; (10) however, the evaluation method has not yet been finalized or made a formal part of the evaluation plan, and questions remain about the definition of terms and small business group measurements; and (11) GAO recognized that further issues could be identified as the evaluation method is refined and finalized.