Specifications of the distributed data storage tool for multiple sources
Authors: Mwinuka, Lunodzo, Cafaro, Massimo
Type: Report
Published: 2026-07-13
Keywords: Data Management, Data Integration, P2P Overlay Networks, Graph Database, Big Data, Distributed Database

<p>This document sets out the specification of the distributed data storage tool for multiple sources in the context of the InnoCyPES project. It is submitted as Deliverable 2.1 and will be periodically reviewed and updated as required. In the InnoCyPES project organisation, Work Package (WP) 2 deals with data accessibility and usability; in particular, this document is concerned with Task 2.1, the design of a distributed data storage service. Therefore, this deliverable provides the design and specifications of the InnoCyPES Data Storage Service, from now on called IDSS. WP 2 will be executed by the University of Salento (UNILE) and Technische Universiteit Delft (TUD). UNILE will deal with data management and integration including ingestion and extraction, whilst TUD will work on data preprocessing to remove the inconsistencies and errors of the data, incorporating physical system knowledge. Tasks 2.1 and 2.2 are hosted by UNILE, Task 2.3 by TUD, as follows: Task 2.1 (UNILE) Design of a distributed data storage service, Task 2.2 (UNILE) Implementation of the distributed data storage service, and Task 2.3 (TUD) Enriching system measurements for machine learning applications.<br>The current landscape of energy systems has undergone significant advancements, resulting in the generation of high-volume and heterogeneous data. This proliferation of data poses challenges in terms of storage, hence constraining businesses' capacity to conduct effective analytics and extract knowledge from them. Traditional storage architectures have proven to be limited in curbing the challenges associated with these trends, sparking a surge of interest in the development of advanced storage systems.<br>Relational databases have historically dominated the database landscape, owing to their simplicity and organised structure. Simultaneously, data warehouses, characterised by centralised architectures, have conventionally managed extensive datasets. However, challenges encountered in the implementation of centrally managed relational databases have spurred the rise of distributed databases and NoSQL solutions. Consequently, contemporary database architectures geared toward handling large-scale datasets focus on the implementation of distributed databases.<br>Within the scope of this document, current approaches have been explored, shedding light on key challenges and proposing optimal deployment strategies for the development and deployment of IDSS. In addressing the issue of a single point of failure, a peer-to-peer distributed architecture is recommended.<br>The main goal of designing a novel database architecture is to maximise system throughput and availability, avoiding issues related to a centralised setting and providing support for complex distributed queries, while also allowing simple and efficient data integration. Consequently, this document puts forth a proposal for an innovative distributed database architecture in the context of the European InnoCyPES project.</p>