Deep GreenAI: Effective Layer Pruning Method for Modern Neural Networks - Pons, Ian; Yamamoto, Bruno; Jordão, Artur (Revista Eletrônica de Iniciação Científica em Computação, 2025)
Deep neural networks have been the predominant paradigm in machine learning for solving cognitive tasks. Such models, however, are restricted by a high computational overhead, limiting their applicability and hindering advancements in the field. Extensive research demonstrated that pruning structures from these models is a straightforward approach to reducing network complexity. In this direction, most efforts focus on removing weights or filters. Studies have also been devoted to layer pruning as it promotes superior computational gains. However, layer removal often hurts network predictive ability (i.e., accuracy) at high compression rates. This work introduces an effective layer-pruning strategy that meets all underlying properties pursued by pruning methods. Our method estimates the relative importance of a layer using the Centered Kernel Alignment (CKA) metric, employed to measure the similarity between representations of the unpruned model and a candidate subnetwork for pruning. We confirm the effectiveness of our method on standard architectures and benchmarks, in which it outperforms existing layer-pruning strategies and other state-of-the-art pruning techniques. Specifically, we remove more than 75% of computation while improving predictive ability and reducing CO2 emissions required for training by 80%, taking an important step towards GreenAI. At higher compression regimes, our method exhibits negligible accuracy drop, while others notably deteriorate it. Apart from these benefits, our pruned models exhibit robustness to adversarial and out-of-distribution samples.