Distributed training of Deep Neural Networks

Abstract

This report provides a comprehensive and unified study of the principles, methods, and systems that enable scalable training and deployment of large-scale models, with a particular focus on parallelism techniques and Mixture of Experts (MoE) architectures. It combines theoretical insights with practical implementations, offering both a conceptual framework and a hands-on perspective on building efficient distributed systems. The second chapter establishes the foundations of distributed machine learning. The third chapter explores core distributed training strategies, focusing on data parallelism and parameter server architectures. It also presents advanced memory optimization techniques, such as Zero Redundancy Optimizer and Fully Sharded Data Parallel. The fourth chapter focuses on model and computational parallelism, including naive model parallelism, pipeline parallelism, and tensor parallelism. The Mixture of Experts (MoE) paradigm represents a major step forward in scaling neural networks and it is covered on chapter five. Unlike traditional dense models, MoE architectures rely on conditional computation, where only a subset of specialized sub-networks, called experts, is activated for each input. To bridge theory and practice, chapter six includes a comprehensive implementation-focused exploration of MoE models. It revisits the transformer architecture, particularly the decoder-only design used in large language models, and demonstrates how MoE layers can be integrated into this framework. Finally, chapter seven is dedicated to advanced parallelism strategies specifically tailored for MoE architectures, where scalability becomes a multi-dimensional optimization problem. It examines how different parallelism techniques, including data, tensor, pipeline, sequence, and expert parallelism, can be combined into sophisticated hybrid configurations.