Resource Utilization of 2D SLAM Algorithms in ROS-Based Systems: an Empirical Evaluation

Engel Hamer; Michel Albonico; Ivano Malavolta

doi:10.5753/jbcs.2025.4343

Authors

Engel Hamer Vrije Universiteit Amsterdam https://orcid.org/0009-0009-9061-6119
Michel Albonico Federal University of Technology of Paraná https://orcid.org/0000-0003-3606-3444
Ivano Malavolta Vrije Universiteit Amsterdam https://orcid.org/0000-0001-5773-8346

DOI:

https://doi.org/10.5753/jbcs.2025.4343

Keywords:

ROS, Software Engineering, Empirical Evaluation, Energy Consumption, SLAM

Abstract

Simultaneous localization and mapping (SLAM) is an important task in robotic systems, which entails mapping an environment while keeping track of the robot's position within the created map. The Robot Operating System (ROS) offers various packages for this functionality, where each one may lead to different performance and resource usage. Therefore, this study aims to investigate the impact of different ROS-based SLAM algorithms on resource utilization, including possible trade-offs with performance (e.g., the accuracy of the created map). The investigation is centered on primary experiments involving multiple runs of a single robot, which alternates between four SLAM algorithms: Cartographer, Gmapping, Hector SLAM, and Karto. During these experiments, the robot autonomously navigates through two types of arenas: point-to-point (multi-goal navigation) and circular (returning to the starting position after following the perimeter). Throughout these trials, the robot's performance is assessed based on the ROS system's efficiency and energy consumption. In a secondary set of experiments, the tests are repeated, but with key SLAM algorithm parameters reconfigured to evaluate their impact. The experiment results reveal Karto as the most efficient algorithm across all evaluated metrics, and the one that creates the most visually consistent maps. Cartographer was the algorithm that showed the least promising results regarding both, energy consumption and CPU utilization. Furthermore, Gmapping was the algorithm most susceptible to changes in SLAM algorithms' parameter values. The results presented in this study are, therefore, key for resource-aware design choices when using SLAM algorithms in the context of ROS-based systems.

Downloads

Download data is not yet available.

References

Abdelrasoul, Y., Saman, A. B. S. H., and Sebastian, P. (2016). A quantitative study of tuning ros gmapping parameters and their effect on performing indoor 2d slam. In 2016 2nd IEEE International Symposium on Robotics and Manufacturing Automation (ROMA), pages 1-6. DOI: 10.1109/ROMA.2016.7847825.

Abdi, H. and Williams, L. J. (2010). Tukey’s honestly significant difference (hsd) test. Encyclopedia of research design, 3(1):1-5. Available online [link].

Aerts, P. and Demeester, E. (2017). Benchmarking of 2d-slam algorithms. In Ad Usum Navigantium. Available online [link].

Albonico, M., Malavolta, I., Pinto, G., Guzman, E., Chinnappan, K., and Lago, P. (2021). Mining energy-related practices in robotics software. In 2021 IEEE/ACM 18th International Conference on Mining Software Repositories (MSR), pages 483-494. DOI: 10.1109/MSR52588.2021.00060.

Alcalde, M., Ferreira, M., González, P., Andrade, F., and Tejera, G. (2022). Da-slam: Deep active slam based on deep reinforcement learning. In 2022 Latin American Robotics Symposium (LARS), 2022 Brazilian Symposium on Robotics (SBR), and 2022 Workshop on Robotics in Education (WRE), pages 282-287. DOI: 10.1109/LARS/SBR/WRE56824.2022.9996006.

Alharbi, M. and Alshayeb, M. (2024). An empirical investigation of the relationship between pattern grime and code smells. Journal of Software: Evolution and Process, page e2666. DOI: 10.1002/smr.2666.

Anderson, M. J. (2001). A new method for non-parametric multivariate analysis of variance. Austral ecology, 26(1):32-46. DOI: 10.1111/j.1442-9993.2001.01070.pp.x.

Andert, F. and Mosebach, H. (2019). Lidar slam positioning quality evaluation in urban road traffic. In International Conference on Intelligent Transport Systems, pages 277-291. Springer. DOI: 10.1007/978-3-030-38822-5_19.

Balaguer, B., Balakirsky, S., Carpin, S., and Visser, A. (2009). Evaluating maps produced by urban search and rescue robots: lessons learned from robocup. Autonomous Robots, 27(4):449-464. DOI: 10.1007/s10514-009-9141-z.

Basili, V., Caldiera, G., and Rombach, H. (1994). Goal question metric paradigm. Encyclopedia of software engineering, 1:528-532. Available online [link].

Benjamini, Y. and Hochberg, Y. (1995). Controlling the false discovery rate: a practical and powerful approach to multiple testing. Journal of the Royal statistical society: series B (Methodological), 57(1):289-300. DOI: 10.1111/j.2517-6161.1995.tb02031.x.

Brown, M. B. and Forsythe, A. B. (1974). Robust tests for the equality of variances. Journal of the American Statistical Association, 69(346):364-367. DOI: 10.1080/01621459.1974.10482955.

Cabane, H. and Farias, K. (2024). On the impact of event-driven architecture on performance: An exploratory study. Future Generation Computer Systems, 153:52-69. DOI: 10.1016/j.future.2023.10.021.

Cadena, C., Carlone, L., Carrillo, H., Latif, Y., Scaramuzza, D., Neira, J., Reid, I., and Leonard, J. (2016). Past, present, and future of simultaneous localization and mapping: Towards the robust-perception age. IEEE Transactions on Robotics, 32(6):1309–1332. DOI: 10.1109/TRO.2016.2624754.

Castaño, J., Martínez-Fernández, S., Franch, X., and Bogner, J. (2023). Exploring the carbon footprint of hugging face's ml models: A repository mining study. In 2023 ACM/IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM), pages 1-12. DOI: 10.1109/ESEM56168.2023.10304801.

Cheeseman, P., Smith, R., and Self, M. (1987). A stochastic map for uncertain spatial relationships. In 4th international symposium on robotic research, pages 467-474. MIT Press Cambridge. Available online [link].

Chong, T., Tang, X., Leng, C., Yogeswaran, M., Ng, O., and Chong, Y. (2015). Sensor technologies and simultaneous localization and mapping (slam). Procedia Computer Science, 76:174-179. DOI: 10.1016/j.procs.2015.12.336.

Costa, L. R. and Colombini, E. L. (2023). Can semantic-based filtering of dynamic objects improve visual slam and visual odometry? In 2023 Latin American Robotics Symposium (LARS), 2023 Brazilian Symposium on Robotics (SBR), and 2023 Workshop on Robotics in Education (WRE), pages 567-572. DOI: 10.1109/LARS/SBR/WRE59448.2023.10332956.

De Mello Gai, A., Bevilacqua, S., Cukla, A. R., and Gamarra, D. F. T. (2023). Evaluation on imu and odometry sensor fusion for a turtlebot robot using amcl on ros framework. In 2023 Latin American Robotics Symposium (LARS), 2023 Brazilian Symposium on Robotics (SBR), and 2023 Workshop on Robotics in Education (WRE), pages 637-642. DOI: 10.1109/LARS/SBR/WRE59448.2023.10332977.

Dhaoui, R. and Rahmouni, A. (2022). Mobile robot navigation in indoor environments: Comparison of lidar-based 2d slam algorithms. In Design Tools and Methods in Industrial Engineering II: Proceedings of the Second International Conference on Design Tools and Methods in Industrial Engineering, ADM 2021, September 9-10, 2021, Rome, Italy, pages 569-580. Springer. DOI: 10.1007/978-3-030-91234-5_57.

Dordevic, M., Hamer, E., and Malavolta, I. (2021). Software engineering research on the robot operating system: A systematic mapping study. DOI: 10.1016/j.jss.2022.111574.

Duchoň, F., Hažík, J., Rodina, J., Tölgyessy, M., Dekan, M., and Sojka, A. (2019). Verification of slam methods implemented in ros. Journal of Multidisciplinary Engineering Science and Technology (JMEST). Available online [link].

Dunn, O. J. (1964). Multiple comparisons using rank sums. Technometrics, 6(3):241-252. DOI: 10.2307/1266041.

Durrant-Whyte, H. and Bailey, T. (2006). Simultaneous localization and mapping: part i. IEEE robotics & automation magazine, 13(2):99-110. DOI: 10.1109/MRA.2006.1638022.

Estefo, P., Simmonds, J., Robbes, R., and Fabry, J. (2019). The robot operating system: Package reuse and community dynamics. Journal of Systems and Software, 151:226-242. DOI: 10.1016/j.jss.2019.02.024.

Filatov, A., Filatov, A., Krinkin, K., Chen, B., and Molodan, D. (2017). 2d slam quality evaluation methods. In 2017 21st Conference of Open Innovations Association (FRUCT), pages 120-126. IEEE. DOI: 10.23919/FRUCT.2017.8250173.

Grisetti, G., Stachniss, C., and Burgard, W. (2007). Improved techniques for grid mapping with rao-blackwellized particle filters. IEEE transactions on Robotics, 23(1):34-46. DOI: 10.1109/TRO.2006.889486.

Harris, C., Stephens, M., et al. (1988). A combined corner and edge detector. In Alvey vision conference, volume 15, pages 10-5244. Citeseer. Available online [link].

Hess, W., Kohler, D., Rapp, H., and Andor, D. (2016). Real-time loop closure in 2d lidar slam. In 2016 IEEE International Conference on Robotics and Automation (ICRA), pages 1271-1278. IEEE. DOI: 10.1109/ICRA.2016.7487258.

Kleiner, A. and Dornhege, C. (2007). Real-time localization and elevation mapping within urban search and rescue scenarios. Journal of Field Robotics, 24(8-9):723-745. DOI: 10.1002/rob.20208.

Kohlbrecher, S., Meyer, J., von Stryk, O., and Klingauf, U. (2011). A flexible and scalable slam system with full 3d motion estimation. In Proc. IEEE International Symposium on Safety, Security and Rescue Robotics (SSRR). IEEE. DOI: 10.1109/SSRR.2011.6106777.

Kolak, S., Afzal, A., Le Goues, C., Hilton, M., and Timperley, C. S. (2020). It takes a village to build a robot: An empirical study of the ros ecosystem. In 2020 IEEE International Conference on Software Maintenance and Evolution (ICSME), pages 430-440. DOI: 10.1109/ICSME46990.2020.00048.

Konolige, K., Grisetti, G., Kümmerle, R., Burgard, W., Limketkai, B., and Vincent, R. (2010). Efficient sparse pose adjustment for 2d mapping. In 2010 IEEE/RSJ International Conference on Intelligent Robots and Systems, pages 22-29. IEEE. DOI: 10.1109/IROS.2010.5649043.

Koziolek, H., Grüner, S., and Rückert, J. (2020). A comparison of mqtt brokers for distributed iot edge computing. In Jansen, A., Malavolta, I., Muccini, H., Ozkaya, I., and Zimmermann, O., editors, Software Architecture, pages 352-368, Cham. Springer International Publishing. DOI: 10.1007/978-3-030-58923-3_23.

Le, X. S., Fabresse, L., Bouraqadi, N., and Lozenguez, G. (2018). Evaluation of out-of-the-box ros 2d slams for autonomous exploration of unknown indoor environments. In Chen, Z., Mendes, A., Yan, Y., and Chen, S., editors, Intelligent Robotics and Applications, pages 283-296, Cham. Springer International Publishing. DOI: 10.1007/978-3-319-97589-4_24.

Lee, S. U., Fernando, N., Lee, K., and Schneider, J.-G. (2024). A survey of energy concerns for software engineering. Journal of Systems and Software, 210:111944. DOI: 10.1016/j.jss.2023.111944.

Lins, F. C. A., Luz, O. M., Medeiros, A. A. D., and Alsina, P. J. (2020). A comparison of three different parameterizations to represent planes for the mapping step of direct visual slam approaches. In 2020 Latin American Robotics Symposium (LARS), 2020 Brazilian Symposium on Robotics (SBR) and 2020 Workshop on Robotics in Education (WRE), pages 1-6. DOI: 10.1109/LARS/SBR/WRE51543.2020.9306932.

Malavolta, I., Chinnappan, K., Swanborn, S., Lewis, G. A., and Lago, P. (2021). Mining the ros ecosystem for green architectural tactics in robotics and an empirical evaluation. In 2021 IEEE/ACM 18th International Conference on Mining Software Repositories (MSR), pages 300-311. IEEE. DOI: 10.1109/MSR52588.2021.00042.

Malavolta, I., Lewis, G., Schmerl, B., Lago, P., and Garlan, D. (2020). How do you architect your robots? state of the practice and guidelines for ros-based systems. In 2020 IEEE/ACM 42nd International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP), pages 31-40. IEEE. DOI: 10.1145/3377813.3381358.

Montemerlo, M., Thrun, S., Koller, D., Wegbreit, B., et al. (2002). Fastslam: A factored solution to the simultaneous localization and mapping problem. Aaai/iaai, 593598. Available online [link].

Nardi, L., Bodin, B., Zia, M. Z., Mawer, J., Nisbet, A., Kelly, P. H. J., Davison, A. J., Luján, M., O'Boyle, M. F. P., Riley, G., Topham, N., and Furber, S. (2015). Introducing slambench, a performance and accuracy benchmarking methodology for slam. In 2015 IEEE International Conference on Robotics and Automation (ICRA), pages 5783-5790. DOI: 10.1109/ICRA.2015.7140009.

Ngoc, H. T., Vinh, N. N., Nguyen, N. T., and Quach, L.-D. (2023). Efficient evaluation of slam methods and integration of human detection with yolo based on multiple optimization in ros2. International Journal of Advanced Computer Science & Applications, 14(11). Available online [link].

Nguyen, Q. H., Johnson, P., and Latham, D. (2022). Performance evaluation of ros-based slam algorithms for handheld indoor mapping and tracking systems. IEEE Sensors Journal, 23(1):706-714. DOI: 10.1109/JSEN.2022.3224224.

Ðorđević, M., Albonico, M., Lewis, G. A., Malavolta, I., and Lago, P. (2023). Computation offloading for ground robotic systems communicating over wifi-an empirical exploration on performance and energy trade-offs. Empirical Software Engineering, 28(6):140. DOI: 10.1007/s10664-023-10351-6.

Partap, A., Grayson, S., Huzaifa, M., Adve, S., Godfrey, B., Gupta, S., Hauser, K., and Mittal, R. (2022). On-device cpu scheduling for robot systems. In 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 11296-11303. DOI: 10.1109/IROS47612.2022.9982085.

Peng, T., Zhang, D., Liu, R., Asari, V. K., and Loomis, J. S. (2019). Evaluating the power efficiency of visual slam on embedded gpu systems. In 2019 IEEE National Aerospace and Electronics Conference (NAECON), pages 117-121. DOI: 10.1109/NAECON46414.2019.9058059.

Quigley, M., Faust, J., Foote, T., Leibs, J., et al. (2009). Ros: an open-source robot operating system. Available online [link].

Roesler, O. and Ravindranath, V. P. (2020). Evaluation of slam algorithms for highly dynamic environments. In Silva, M. F., Luís Lima, J., Reis, L. P., Sanfeliu, A., and Tardioli, D., editors, Robot 2019: Fourth Iberian Robotics Conference, pages 28-36, Cham. Springer International Publishing. DOI: 10.1007/978-3-030-36150-1_3.

Rojas-Fernández, M., Mújica-Vargas, D., Matuz-Cruz, M., and López-Borreguero, D. (2018). Performance comparison of 2d slam techniques available in ros using a differential drive robot. In 2018 International Conference on Electronics, Communications and Computers (CONIELECOMP), pages 50-58. DOI: 10.1109/CONIELECOMP.2018.8327175.

Samarakoon, K. Y., Pereira, G. A. S., and Gross, J. N. (2022). Impact of the trajectory on the performance of rgb-d slam executed by a uav in a subterranean environment. In 2022 International Conference on Unmanned Aircraft Systems (ICUAS), pages 812-820. DOI: 10.1109/ICUAS54217.2022.9836199.

Sankalprajan, P., Sharma, T., Perur, H. D., and Sekhar Pagala, P. (2020). Comparative analysis of ros based 2d and 3d slam algorithms for autonomous ground vehicles. In 2020 International Conference for Emerging Technology (INCET), pages 1-6. DOI: 10.1109/INCET49848.2020.9154101.

Santos, J. M., Portugal, D., and Rocha, R. P. (2013). An evaluation of 2d slam techniques available in robot operating system. In 2013 IEEE International Symposium on Safety, Security, and Rescue Robotics (SSRR), pages 1-6. IEEE. DOI: 10.1109/SSRR.2013.6719348.

Shapiro, S. S. and Wilk, M. B. (1965). An analysis of variance test for normality (complete samples). Biometrika, 52(3/4):591-611. DOI: 10.1093/biomet/52.3-4.591.

Smith, R. C. and Cheeseman, P. (1986). On the representation and estimation of spatial uncertainty. The international journal of Robotics Research, 5(4):56-68. DOI: 10.1177/027836498600500404.

Snedecor, G. W., Cochran, W. G., et al. (1968). Statistical methods. Number 6th ed.(repr.). Iowa state university press. Book.

Stachniss, C., Leonard, J. J., and Thrun, S. (2016). Simultaneous localization and mapping. In Springer Handbook of Robotics, pages 1153-1176. Springer. DOI: 10.1007/978-3-319-32552-1_46.

Sturm, J., Engelhard, N., Endres, F., Burgard, W., and Cremers, D. (2012). A benchmark for the evaluation of rgb-d slam systems. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pages 573-580. DOI: 10.1109/IROS.2012.6385773.

Su, Z., Zhou, J., Dai, J., and Zhu, Y. (2020). Optimization design and experimental study of gmapping algorithm. In 2020 Chinese Control And Decision Conference (CCDC), pages 4894-4898. DOI: 10.1109/CCDC49329.2020.9164603.

Suger, B., Tipaldi, G. D., Spinello, L., and Burgard, W. (2014). An approach to solving large-scale slam problems with a small memory footprint. 2014 IEEE International Conference on Robotics and Automation (ICRA), pages 3632-3637. DOI: 10.1109/ICRA.2014.6907384.

Suzuki, S. et al. (1985). Topological structural analysis of digitized binary images by border following. Computer vision, graphics, and image processing, 30(1):32-46. DOI: 10.1016/0734-189X(85)90016-7.

Swanborn, S. (2020). An empirical evaluation of energy-efficient architectural tactics in robotics software.

Swanborn, S. and Malavolta, I. (2021). Robot Runner: A Tool for Automatically Executing Experiments on Robotics Software. In Proceedings of the ACM/IEEE 43rd International Conference on Software Engineering. ACM. DOI: 10.1109/ICSE-Companion52605.2021.00029.

Thrun, S. et al. (2002). Robotic mapping: A survey. Available online [link].

Vaussard, F., Rétornaz, P., Hamel, D., and Mondada, F. (2012). Cutting down the energy consumed by domestic robots: Insights from robotic vacuum cleaners. In Advances in Autonomous Robotics, pages 128-139, Berlin, Heidelberg. Springer Berlin Heidelberg. DOI: 10.1007/978-3-642-32527-4_12.

Wohlin, C., Runeson, P., Host, M., Ohlsson, M., Regnell, B., and Wesslen, A. (2012). Experimentation in software engineering - an introduction. DOI: 10.1007/978-3-662-69306-3.

Yagfarov, R., Ivanou, M., and Afanasyev, I. (2018). Map comparison of lidar-based 2d slam algorithms using precise ground truth. In 2018 15th International Conference on Control, Automation, Robotics and Vision (ICARCV), pages 1979-1983. DOI: 10.1109/ICARCV.2018.8581131.

Zhao, Y.-L., Hong, Y.-T., and Huang, H.-P. (2024). Comprehensive performance evaluation between visual slam and lidar slam for mobile robots: Theories and experiments. Applied Sciences, 14(9):3945. DOI: 10.3390/app14093945.

Zia, M. Z., Nardi, L., Jack, A., Vespa, E., Bodin, B., Kelly, P. H., and Davison, A. J. (2016). Comparative design space exploration of dense and semi-dense slam. In 2016 IEEE International Conference on Robotics and Automation (ICRA), pages 1292-1299. DOI: 10.1109/ICRA.2016.7487261.