Learning Parallel Computing with Multilayer Perceptron Neural Networks

Abstract

As multi-core architectures become the standard for computational efficiency, parallel programming has evolved into a primary avenue for optimization and resource management. However, an integrated design-and-evaluation method is often difficult for beginners to envision, leading to incorrect intuition about the relationship between theory and practice. This article proposes an implementation-driven methodological approach using a Multilayer Perceptron (MLP) to bridge this gap, comprising three main components: (1) establishing a sequential baseline, (2) applying parallelization and data containerization strategies to the MLP, and (3) evaluating through key metrics such as speedup and hardware efficiency. By addressing data dependencies and synchronization challenges inherent to backpropagation, the proposed trajectory provides a framework for comprehending data integrity in parallel systems through an exemplified parallel implementation. Our approach led to successful optimization of the feedforward and backpropagation routines, resulting in a speedup, e.g., of 4 times the baseline execution time using 8 hardware threads. Furthermore, the results indicate that outer-loop nesting and data containerization should be considered for managing structures with high data dependencies.