using observational pilot studies to test and improve lab packages
Controlled experiments are a key approach to evaluate and evolve our understanding of software engineering technologies. However, defining and running a controlled experiment is a difficult and errorprone task. This paper argues that one can significantly reduce the risks associated with defining a new controlled experiment by running a set of well-planned observational pilot studies aimed at improving the experimental material. It gives the steps for such an approach and illustrates it with a case study that shows the definition of an experiment to evaluate inspection techniques for information visualization tools through a set of four observational studies and one experimental trial. Based on the lessons learned we present some guidelines on how to test and improve experimental material in such way.
lab package inspection techniques empirical studies mining visual data