Exploring Normalization for High Convergence on Federated Learning for Drones
DOI:
https://doi.org/10.5753/jbcs.2024.4133Keywords:
Federated Learning, Heterogeneous Data, Edge Devices, Data Transformation, Image ClassificationAbstract
The usage of mobile devices like drones has been increasing in various fields, ranging from package delivery to emergency services and environmental monitoring. Intelligent services increasingly use the processing power of these devices in conjunction with techniques such as Federated Learning (FL), which allows machine learning to be carried out in a decentralized way using data accessed by clients or devices. However, in normal operations, the data accessed by clients is distributed heterogeneously among themselves, negatively impacting learning results. This article discusses the normalization in Federated Learning local training to mitigate results obtained in heterogeneous distributions. In this context, we propose Federated Learning with Weight Standardization on Convolutional Neural Networks (FedWS) and evaluate it with Batch Normalization, Layer Normalization, and Group Normalization in experiments with heterogeneous data distributions. The experiments demonstrated that FedWS achieved higher accuracy results ranging from 3% to 6% and reduced the computational and communication costs between 25% and 40%, being more suitable for use in devices with computational resource limitations.
Downloads
References
Alsamhi, S. H., Shvetsov, A. V., Kumar, S., Hassan, J., Alhartomi, M. A., Shvetsova, S. V., Sahal, R., and Hawbani, A. (2022). Computing in the sky: A survey on intelligent ubiquitous computing for uav-assisted 6g networks and industry 4.0/5.0. Drones, 6(7):177. DOI: 10.3390/drones6070177.
Asad, M., Moustafa, A., Ito, T., and Aslam, M. (2021). Evaluating the communication efficiency in federated learning algorithms. In 24th IEEE International Conference on Computer Supported Cooperative Work in Design (CSCWD), pages 552-557. DOI: 10.1109/CSCWD49262.2021.9437738.
Butler, L., Yigitcanlar, T., and Paz, A. (2020). Smart urban mobility innovations: A comprehensive review and evaluation. IEEE ACCESS, 8:196034-196049. DOI: 10.1109/ACCESS.2020.3034596.
Causa, F., Franzone, A., and Fasano, G. (2023). Strategic and tactical path planning for urban air mobility: Overview and application to real-world use cases. Drones, 7(1):11. DOI: 10.3390/drones7010011.
Du, Z., Sun, J., Li, A., Chen, P.-Y., Zhang, J., Li, H. H., and Chen, Y. (2022). Rethinking normalization methods in federated learning. In 3rd International Workshop on Distributed Machine Learning, pages 16-22. DOI: 10.1145/3565010.3569062.
Duan, Q., Huang, J., Hu, S., Deng, R., Lu, Z., and Yu, S. (2023). Combining federated learning and edge computing toward ubiquitous intelligence in 6g network: Challenges, recent advances, and future directions. IEEE Communications Surveys & Tutorials. DOI: 10.1109/COMST.2023.3316615.
Helber, P., Bischke, B., Dengel, A., and Borth, D. (2019). Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 12(7):2217-2226. DOI: 10.1109/JSTARS.2019.2918242.
Ioffe, S. and Szegedy, C. (2015). Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International Conference on Machine Learning (ICML), pages 448-456. Available online [link].
Khan, L. U., Saad, W., Han, Z., Hossain, E., and Hong, C. S. (2021). Federated learning for internet of things: Recent advances, taxonomy, and open challenges. IEEE Communications Surveys & Tutorials, 23(3):1759-1799. DOI: 10.1109/COMST.2021.3090430.
LeCun, Y. et al. (2015). Lenet-5, convolutional neural networks. URL: http://yann. lecun. com/exdb/lenet, 20(5):14. Available online [link].
Li, Q., Diao, Y., Chen, Q., and He, B. (2022). Federated learning on non-iid data silos: An experimental study. In 38th IEEE International Conference on Data Engineering (ICDE), pages 965-978. DOI: 10.1109/ICDE53745.2022.00077.
Li, X., Jiang, M., Zhang, X., Kamp, M., and Dou, Q. (2021). Fedbn: Federated learning on non-iid features via local batch normalization. arXiv preprint arXiv:2102.07623. DOI: 10.48550/arXiv.2102.07623.
Liu, S., Yu, J., Deng, X., and Wan, S. (2021). Fedcpf: An efficient-communication federated learning approach for vehicular edge computing in 6g communication networks. IEEE Transactions on Intelligent Transportation Systems, 23(2):1616-1629. DOI: 10.1109/TITS.2021.3099368.
Lubana, E. S., Dick, R., and Tanaka, H. (2021). Beyond batchnorm: towards a unified understanding of normalization in deep learning. Advances in Neural Information Processing Systems, 34:4778-4791. DOI: 10.48550/arXiv.2106.05956.
McMahan, B., Moore, E., Ramage, D., Hampson, S., and y Arcas, B. A. (2017). Communication-efficient learning of deep networks from decentralized data. In Artificial intelligence and statistics, pages 1273-1282. DOI: 10.48550/arXiv.1602.05629.
Qiao, S., Wang, H., Liu, C., Shen, W., and Yuille, A. (2019). Micro-batch training with batch-channel normalization and weight standardization. arXiv preprint arXiv:1903.10520. DOI: 10.48550/arXiv.1903.10520.
Rodríguez-Barroso, N., Jiménez-López, D., Luzón, M. V., Herrera, F., and Martínez-Cámara, E. (2023). Survey on federated learning threats: Concepts, taxonomy on attacks and defences, experimental study and challenges. Information Fusion, 90:148-173. DOI: 10.1016/j.inffus.2022.09.011.
Santurkar, S., Tsipras, D., Ilyas, A., and Madry, A. (2018). How does batch normalization help optimization? Advances in neural information processing systems, 31. DOI: 10.48550/arXiv.1805.11604.
Shome, D., Waqar, O., and Khan, W. U. (2022). Federated learning and next generation wireless communications: A survey on bidirectional relationship. Transactions on Emerging Telecommunications Technologies, 33(7):e4458. DOI: 10.1002/ett.4458.
Vieira, F. and Campos, C. A. V. (2023). Fedws: Uma nova abordagem para aprendizado federado usando dados heterogêneos. In Anais do XXII Workshop em Desempenho de Sistemas Computacionais e de Comunicação, pages 1-12. SBC. DOI: 10.5753/wperformance.2023.230814.
Wen, J., Zhang, Z., Lan, Y., Cui, Z., Cai, J., and Zhang, W. (2023). A survey on federated learning: challenges and applications. International Journal of Machine Learning and Cybernetics, 14(2):513-535. DOI: 10.1007/s13042-022-01647-y.
Wu, Y. and He, K. (2018). Group normalization. In European Conference on Computer Vision (ECCV), pages 3-19. Available online [link].
Xu, Y., Liao, Y., Xu, H., Ma, Z., Wang, L., and Liu, J. (2022). Adaptive control of local updating and model compression for efficient federated learning. IEEE Transactions on Mobile Computing. DOI: 10.1109/TMC.2022.3186936.
Yazid, Y., Ez-Zazi, I., Guerrero-Gonzalez, A., El Oualkadi, A., and Arioua, M. (2021). Uav-enabled mobile edge-computing for iot based on ai: A comprehensive review. Drones, 5(4):148. DOI: 10.3390/drones5040148.
Yu, F., Zhang, W., Qin, Z., Xu, Z., Wang, D., Liu, C., Tian, Z., and Chen, X. (2021). Fed2: Feature-aligned federated learning. In 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, pages 2066-2074. DOI: 10.1145/3447548.3467309.
Zhang, Z., Yang, Y., Yao, Z., Yan, Y., Gonzalez, J. E., Ramchandran, K., and Mahoney, M. W. (2021). Improving semi-supervised federated learning by reducing the gradient diversity of models. In 2021 IEEE International Conference on Big Data (Big Data), pages 1214-1225. IEEE. DOI: 10.1109/BigData52589.2021.9671693.
Zhao, Y., Li, M., Lai, L., Suda, N., Civin, D., and Chandra, V. (2018). Federated learning with non-iid data. arXiv preprint arXiv:1806.00582. DOI: 10.48550/arXiv.1806.00582.
Zhu, H., Xu, J., Liu, S., and Jin, Y. (2021). Federated learning on non-iid data: A survey. Neurocomputing, 465:371-390. DOI: 10.1016/j.neucom.2021.07.098.
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2024 Flavio Vieira, Carlos Alberto V. Campos
This work is licensed under a Creative Commons Attribution 4.0 International License.
