Conceptual formalization of massive storage for advancing decision-making with data analytics

Abstract

Data Lakes have been widely used to handle massive amounts of data arriving at high velocity and variety. However, if proper data management concerns are not addressed, this massive data storage can easily turn Data Lakes into Data Swamps. Furthermore, data must be associated with the data artefacts created to extract value from it, such as pipelines used to collect, treat, or process data and analytical artefacts such as analytical dashboards and machine learning models. This paper proposes a more comprehensive view of a Data Lake, in which all of these resources can be stored and managed. To that end, the conceptual meta-model incorporates a data catalog, data at various stages of maturity, pipelines, dashboards, and machine learning models. The proposed meta-model was instantiated in the ADM.IN (Advanced Decision Making in Productive Systems through Intelligent Networks) project, showing how vast amounts of data and their related artefacts can be managed to support decision-making processes with data analytics.