tion Language (WSDL) (Haas & Brown, 2004) . In the context of SOA, Web services can be implemented and owned by one organization, and published as an independent resource that is consumed by other organizations. To implement complex applications, Web services have to be loosely orchestrated to fulfill a business goal (Peltz, 2003; Sun, Khoury, & Aiello, 2011) . Let us consider an e-bookstore system constructed by several Web services. Among them, a Web service is responsible for the electronic payment. Usually, such a Web service is developed and owned by a third-party organization, such as a software company, a bank, or an independent commercial office. Due to the fact that the consumer (i.e., the ebookstore system) can access the Web service only through its description (namely WSDL) and cannot look into the source code of the Web service, this results in some inconsistency issues. For example, some faults may exist in the implementation regardless how many efforts are spent on testing. Also, the service owner may update the implementation due to changes of the payment policy (specification); however, the service consumer may not realize the changes happening to the implementation or specification. This may result in the situation where the consumer invokes the latest version of implementation while holds the old version of specification. All these cases bring us one question, namely "how should we assure the consistency between implementation and specification of Web services?" Software testing provides a practical and feasible approach to the question. The basic idea of testing is to select some test cases for executing program under test, and then evaluate the output against the expected correct output (namely oracle). If the actual output equals to the oracle, such a test succeeds; otherwise the test detects a fault. In the context of SOA, new unique features pose challenges for testing Web services. For instance, the specification and implementation of Web services are completely separated, and often service consumers cannot access the source codes of Web services they invoke. The lack of source code and the restricted control of Web services not only limit the testability of Web services, but also make white-box testing techniques inapplicable. One may argue that the lack of source codes is also present in the use of any normal software library, but they are totally different in that the implementation of a Web service can be written in any programming language and its specification is written in the XML style. Moreover, the test oracle problem, which means in some situations it is impossible or practically too difficult to decide whether the program outputs on test cases are correct (Weyuker, 1982) , is even amplified. In the example of the electronic payment service, the consumer, in some situations, may not know exactly how much should be charged for a given input (i.e., book price). The problem becomes even worse when the payment involves charges for transfer between accounts or currency exchange. Testing Web service under SOA calls for new techniques (Bartolini, Bertolino, Elbaum, & Marchetti, 2009; Canfora & Penta, 2009; Farooq, Georgieva, & Dumke, 2008; Sun, 2011) . Although there exist many techniques for Web service testing in the literature , Bai, Lee et al. 2008 , Lenz, Chimiak-opoka et al. 2007 , Zhang and Qiu 2006 , most of these existing testing techniques always assume the presence of test oracles and thus cannot address the oracle problem. Metamorphic Testing (MT) was first proposed by Chen,Cheung et al. (1998) as a testing approach without the need of oracles. The basic idea behind MT is to leverage the metamorphic property of Web services to generate test cases and evaluate test outputs. Metamorphic property is a kind of inherent properties of software, referring to that for a pair of test cases satisfying relation R, and then their corresponding outputs should also satisfy relation R f . During the testing, if there exist some tests where R is satisfied while R f is violated, then a fault is detected. In this way, one can employ MT to test programs without precise oracles. It has been shown that MT has successfully alleviated the test oracle problem (Dong, Xu, Chen, Nie, & Wang, 2009 ). However, new features of SOA prevent the traditional MT framework from testing Web services. Recall the e-bookstore International Journal of Web Services Research, 9(1), 51-73, January-March 2012 53 Copyright © 2012, IGI Global. Copying or distributing in print or electronic forms without written permission of IGI Global is prohibited. system, the electronic payment Web services may be deployed in a different namespace with the client program, and it is invoked by a standard protocol (such as SOAP). This results in differences between testing a Web service and traditional software with libraries. This paper proposes a metamorphic relation-based approach to testing Web services to address the challenges of testing Web services under SOA, in particular relevant to the prominent oracle problem with testing Web services. We propose a framework to examine key issues when applying MT to the testing of Web services, and developed a prototype to partially automate the framework. To validate the feasibility and effectiveness of the proposed approach, we conducted empirical studies where three real-life Web services are tested using our approach. Experimental results show that our approach can not only alleviate the test oracle problem of testing Web services, but also deliver an effective and efficient test technique without oracles. The work in this paper makes the following contributions: 1. A MT framework which examines and answers the key issues when using MT to test Web services, and a prototype which partially automates the framework. 2. An efficient testing technique for Web services. As to be observed from the results of the mutation analysis, MT can detect up to 94.1% faults of the subject Web services without oracles. 3. Empirical studies which describe how MT can be employed to test three real-life and widely-practiced Web services and report the effectiveness. Results of the empirical studies clearly show the applicability and effectiveness of MT for testing Web services. The rest of the paper is organized as follows. Next section introduces underlying concepts related to MT and mutation analysis, and then we propose our approach including a framework of MT for Web services and a prototype. After that, we report empirical studies where the proposed approach is employed to test three real-life Web services, followed by a brief comparison of our approach with relevant methods. Finally, we conclude the paper and point out the future work.
Sun et al. proposed an MT framework for web services and conducted a case study on a transfer function of a bank system REF .
A Metamorphic Relation-Based Approach to Testing Web Services Without Oracles
Int. J. Web Service Res.
Int. J. Web Service Res.
Computer Science