Abstract. DIRAC, the LHCb community Grid solution, was considerably reengineered in order to meet all the requirements for processing the data coming from the LHCb experiment. It is covering all the tasks starting with raw data transportation from the experiment area to the grid storage, data processing up to the final user analysis. The reengineered DIRAC3 version of the system includes a fully grid security compliant framework for building service oriented distributed systems; complete Pilot Job framework for creating efficient workload management systems; several subsystems to manage high level operations like data production and distribution management. The user interfaces of the DIRAC3 system providing rich command line and scripting tools are complemented by a full-featured Web portal providing users with a secure access to all the details of the system status and ongoing activities. We will present an overview of the DIRAC3 architecture, new innovative features and the achieved performance. Extending DIRAC3 to manage computing resources beyond the WLCG grid will be discussed. Experience with using DIRAC3 by other user communities than LHCb and in other application domains than High Energy Physics will be shown to demonstrate the general-purpose nature of the system.  The LHCb Collaboration is constructing and operating one of the four experiments running on the LHC proton-proton collider at CERN, Geneva Switzerland. The DIRAC project was originally started in 2002 as a tool to support a large-scale production of the Monte-Carlo experiment modelling data for the LHCb Collaboration. The main design goals were defined as the following: • Transparent access to heterogeneous computing resources available to LHCb at that time. These included various computing clusters in the LHCb collaborating institutes, as well as computing facilities at CERN but also individual workstations of the LHCb users. • An integrated production system with minimal manpower requirements for its maintenance and operation. In particular, no special support by the site managers was considered to be a mandatory feature. After a quick prototyping phase, the DIRAC project was successfully used in production in 2003 and in subsequent productions with resources provided both by the LCG computing grid and independent computing clusters (called DIRAC sites). We would like to note that the LHCb data production run in summer 2004 was the first successful demonstration of the massive usage of the LCG grid resources. This was largely due to the use of the DIRAC Workload Management System, which boosted significantly the efficiency of the production jobs. Based on the successful experience with the DIRAC usage for the Monte-Carlo data production, it was decided to extend the system to support also Data Processing and User Analysis activities. This resulted in the development of the second generation of the DIRAC project software that was now addressing most of the LHCb distributed computing tasks. The successful demonstration of the DIRAC functionality and performance allowed adopting it as the basis of the LHCb distributed computing system in the Computing TDR [1] . In order to deliver the highest quality product by the time of the LHC operation, it was decided that the project should undergo a profound review of its architecture and design solutions. The review took place in the end 2005 -beginning 2006 with multiple recommendations taken on board by the project development team. As a result, the DIRAC software was completely reengineered in order to meet the overview conclusions. A new, third generation of DIRAC was rolled out in summer 2008 and progressively put in production. Now the DIRAC project provides a complete set of tools to support the whole LHCb data processing chain starting from Monte-Carlo Data Production and real Data Distribution to the final end User Analysis tasks. It is important to note that all the DIRAC facilities are implemented within the same base Framework by a single concerted team of developers sharing basic development tools and design solutions. This allowed a relatively small group of developers to deliver quickly a comprehensive suit of tools covering all the needs in the distributed computing management of such a large community as the LHCb Collaboration. In this paper we will discuss the details and the outcome of the DIRAC Review in Section 2. The new features of the latest DIRAC project generation are covered in the subsequent sections. In Section 3 we present the new refurbished Framework. The new features of the Workload Management System are described in Section 4. In DIRAC much attention is paid to failure resilience, these aspects are discussed in Section 5. Since the DIRAC system is used with an increasingly high number of nonexpert users, the importance of the user-friendly interfaces became evident. The new Web Portal allowing to monitor and control all the DIRAC activities is presented in Section 6. A higher-level Production System and other services based on the DIRAC Framework are outlined in Section 7. A summary of the operational experience can be found in Section 8. In Section 9 we present the conclusions and outlook for the future developments.
DIRAC REF is an example of such a system.
DIRAC3 – the new generation of the LHCb grid software
Journal of Physics: Conference Series
Journal of Physics: Conference Series
Computer Science