Accessibility Mutation Testing of Android Applications
DOI:
https://doi.org/10.5753/jserd.2022.2133Keywords:
Mobile Apps, Mutation Testing, AccessibilityAbstract
Smart devices and their apps are present in many everyday activities and play an important role for people with some disabilities. However, making apps more accessible is still a challenge for developers. Automatically accessibility testing tools can help in this task but present some limitations. They produce reports on accessibility faults, which usually cover only a subset of the app because they are dependent on the test set available. In order to help in the improvement and/or assessment of test suites generated, as well as contribute to increasing the performance of accessibility testing tools, this work introduces a mutation testing approach. The approach includes a set of mutant operators derived from faults corresponding to the negation of the WCAG standard’s principles and success criteria. It also includes a process to analyse the mutants regarding the original app. Evaluation results with 7 open-source apps show the approach is applicable in practice and contributes to significantly improving the number of faults revealed by the test suites accompanying the apps.
Downloads
References
Abuaddous, H. Y., Jali, M. Z., and Basir, N. (2016). Web accessibility challenges. International Journal of Advanced Computer Science and Applications (IJACSA).
Acosta-Vargas, P., Salvador-Ullauri, L., Jadán-Guerrero, J., Guevara, C., Sanchez-Gordon, S., Calle-Jimenez, T., LaraAlvarez, P., Medina, A., and Nunes, I. L. (2020). Accessibility assessment in mobile applications for android. In Nunes, I. L., editor, Advances in Human Factors and Systems Interaction, pages 279–288, Cham. Springer International Publishing.
Acosta-Vargas, P., Salvador-Ullauri, L., Perez Medina, J. L., Zalakeviciute, R., and Perdomo, W. (2019). Heuristic method of evaluating accessibility of mobile in selected applications for air quality monitoring. In International Conference on Applied Human Factors and Ergonomics, pages 485–495. Springer.
Alshayban, A., Ahmed, I., and Malek, S. (2020). Accessibility issues in android apps: State of affairs, sentiments, and ways forward. In Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering, ICSE’20, page 1323–1334, New York, NY, USA. Association for Computing Machinery.
Ballantyne, M., Jha, A., Jacobsen, A., Hawker, J. S., and ElGlaly, Y. N. (2018). Study of Accessibility Guidelines of Mobile Applications. In Proceedings of the 17th Interna tional Conference on Mobile and Ubiquitous Multimedia, pages 305–315. ACM.
BBC (2017). The BBC Standards and Guidelines for Mobile Accessibility. [Link].
Brazilian Government (2007). Accessibility Model in Electronic Government. [Link].
Cisco (2017). Cisco Visual Networking Index: Global Mobile Data Traffic Forecast Update, 2017–2022 White Paper - Cisco. [Link].
Damaceno, R. J. P., Braga, J. C., and Mena-Chalco, J. P. (2018). Mobile device accessibility for the visually impaired: problems mapping and recommendations. Universal Access in the Information Society, 17(2):421–435.
Delamaro, M. E., Offutt, J., and Ammann, P. (2014). Designing deletion mutation operators. In 2014 IEEE Seventh International Conference on Software Testing, Verification and Validation, pages 11–20.
Deng, L., Mirzaei, N., Ammann, P., and Offutt, J. (2015). Towards mutation analysis of Android apps. In Proceedings of the Eighth International Conference on Software Testing, Verification and Validation Workshops, ICSTW, pages 1–10. IEEE.
Eler, M. M., Rojas, J. M., Ge, Y., and Fraser, G. (2018). Automated Accessibility Testing of Mobile Apps. In 2018 IEEE 11th International Conference on Software Testing, Verification and Validation (ICST), pages 116–126.
Escobar-Velásquez, Camilo, O.-R., Michael, and LinaresVásquez, M. (2019). MutAPK: Source-Codeless Mutant Generation for Android Apps. In 2019 IEEE/ACM International Conference on Automated Software Engineering (ASE).
Gamma, E. and Beck, K. (2019). The new major version of the programmer-friendly testing framework for Java. [Link].
Google (2018). Espresso. [Link].
Google (2018). Improve your code with lint checks. [Link].
Google (2020). Accessibility Scanner. [Link].
Grechanik, M., Xie, Q., and Fu, C. (2009). Creating gui testing tools using accessibility technologies. In 2009 International Conference on Software Testing, Verification, and Validation Workshops, pages 243–250.
Gu, T., Sun, C., Ma, X., Cao, C., Xu, C., Yao, Y., Zhang, Q., Lu, J., and Su, Z. (2019). Practical GUI Testing of Android Applications via Model Abstraction and Refinement. In Proceedings of the 41st International Conference on Software Engineering, ICSE ’19, page 269–280. IEEE Press.
Hartley, S. D. (2011). World Report on Disability (WHO). Technical report, WHO and World Bank.
Jabbarvand, R. and Malek, S. (2017). µDroid: an energyaware mutation testing framework for Android. In Proceedings of the 11th Joint Meeting on Foundations of Software Engineering, ESEC/FSE, pages 208–219. ACM.
Jia, Y. and Harman, M. (2011). An analysis and survey of the development of mutation testing. IEEE Trans. Software Eng., 37(5):649–678.
Kirkpatrick, A., Connor, J. O., Campbell, A., and Cooper, M. (2018). Web Content Accessibility Guidelines (WCAG) 2.1. [Link].
Linares-Vásquez, M., Bavota, G., Tufano, M., Moran, K., Di Penta, M., Vendome, C., Bernal-Cárdenas, C., and Poshyvanyk, D. (2017). Enabling Mutation Testing for Android Apps. In Proceedings of the 2017 11th Joint Meeting on Foundations of Software Engineering, ESEC/FSE, pages 233–244, New York, NY, USA. ACM.
Lisper, B., Lindstrom, B., Potena, P., Saadatmand, M., and Bohlin, M. (2017). Targeted mutation: Efficient mutation analysis for testing non-functional properties. In Proceedings - 10th IEEE International Conference on Software Testing, Verification and Validation Workshops, (ICSTW), pages 65–68.
Luna, E. and El Ariss, O. (2018). Edroid: A Mutation Tool for Android Apps. In Proceedings of the 6th International Conference in Software Engineering Research and Innovation, CONISOFT, pages 99–108. IEEE.
Mao, K., Harman, M., and Jia, Y. (2016). Sapienz: Multiobjective automated testing for android applications. In Proceedings of the 25th International Symposium on Software Testing and Analysis, ISSTA 2016, page 94–105, New York, NY, USA. Association for Computing Machinery.
Moher, D., Liberati, A., Tetzlaff, J., and Altman, D. G. (2009). Preferred Reporting Items for Systematic Reviews and Meta-Analyses: The PRISMA Statement. BMJ, 339.
Moran, K., Tufano, M., Bernal-Cárdenas, C., LinaresVásquez, M., Bavota, G., Vendome, C., Di Penta, M., and Poshyvanyk, D. (2018). Mdroid+: A mutation testing framework for android. In Proceedings of the 40th International Conference on Software Engineering: Companion Proceeedings, pages 33–36. ACM.
Reda, R. (2019). RobotiumTech: Android UI Testing. [Link].
SejaSidier (2015). Guide to the Development of Accessible Mobile Applications. [Link].
Silva, C., Eler, M. M., and Fraser, G. (2018). A survey on the tool support for the automatic evaluation of mobile accessibility. In Proceedings of the 8th International Conference on Software Development and Technologies for Enhancing Accessibility and Fighting Info-Exclusion, DSAI 2018, page 286–293. ACM.
Silva, H. N., Endo, A. T., Eler, M. M., Vergilio, S. R., and Durelli, V. H. R. (2020). On the Relation between Code Elements and Accessibility Issues in Android Apps. In Proceedings of the V Brazilian Symposium on Systematic and Automated Software Testing, SAST.
Silva, H. N., Prado Lima, J. A., Endo, A. T., and Vergilio, S. R. (2021). A mapping study on mutation testing for mobile applications. Software Testing, Verification Reliability.
Su, T., Meng, G., Chen, Y., Wu, K., Yang, W., Yao, Y., Pu, G., Liu, Y., and Su, Z. (2017). Guided, stochastic modelbased GUI testing of android apps. In Proceedings of the 11th Joint Meeting on Foundations of Software Engineering, ESEC/FSE, Paderborn, Germany, September 4-8, pages 245–256.
Toff, D. (2018). A11y ally. [Link].
Vendome, C., Solano, D., Liñán, S., and Linares-Vásquez, M. (2019). Can Everyone use my app? An Empirical Study on Accessibility in Android Apps. In 2019 IEEE International Conference on Software Maintenance and Evolution (ICSME), pages 41–52.
W3C (2019). W3C Accessibility Standards Overview. [Link].
Wei, Y. (2015). MuDroid: Mutation Testing for Android Apps. Technical report, UCL-UK. Undergraduate Final Year Individual Project.
Wille, K., Dumke, R. R., and Wille, C. (2016). Measuring the accessability based on web content accessibility guidelines. In 2016 Joint Conference of the International Workshop on Software Measurement and the International Conference on Software Process and Product Measurement (IWSM-MENSURA), pages 164–169.
Yan, S. and Ramachandran, P. G. (2019). The current status of accessibility in mobile apps. ACM Transactions on Accessible Computing, 12.
Zeng, X., Li, D., Zheng, W., Xia, F., Deng, Y., Lam, W., Yang, W., and Xie, T. (2016). Automated test input generation for android: Are we really there yet in an industrial case? In Proceedings of the 2016 24th ACM SIGSOFT International Symposium on Foundations of Software Engineering, FSE 2016, page 987–992.
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2022 Henrique Neves da Silva, Silvia Regina Vergilio, André Takeshi Endo
This work is licensed under a Creative Commons Attribution 4.0 International License.
