Advancing data architectures with Data Mesh implementations

Abstract

Data architectures have evolved over time to adapt to the growing needs of the business models. Recently, the Data Mesh concept emerged due to the limitations of current monolithic data architectures and the investments they require. It represents a paradigm shift in the way data architectures are thought and work, applying the microservices logic of software engineering to data engineering, and intends to make organizations truly data-driven, with data becoming the primary concern, leaving the data pipelines to secondary plan. These characteristics imply a change not only in the technological part of the data architecture, but also in the way the data teams are organized. Due to the youthfulness of the topic, it still lacks scientific foundation on the premises that are associated with it. Contributing to those foundations, this paper proposes a technological architecture for the implementation of a Data Mesh and evaluates the proposal with a demonstration case that highlights the usefulness and benefits of this type of data architecture.