A Systematic Mapping of Accessibility Problems Encountered on Websites and Mobile Apps: A Comparison Between Automated Tests, Manual Inspections and User Evaluations

Delvani Antônio Mateus; Carlos Alberto Silva; Arthur F. B. A. de Oliveira; Heitor Costa; André Pimenta Freire

doi:10.5753/jis.2021.1778

Authors

Delvani Antônio Mateus Universidade Federal de Lavras https://orcid.org/0000-0002-5832-3115
Carlos Alberto Silva Universidade Federal de Lavras https://orcid.org/0000-0002-3535-0065
Arthur F. B. A. de Oliveira Universidade de São Paulo https://orcid.org/0000-0002-3828-4331
Heitor Costa Universidade Federal de Lavras https://orcid.org/0000-0002-9903-7414
André Pimenta Freire Universidade Federal de Lavras https://orcid.org/0000-0001-7894-9740

DOI:

https://doi.org/10.5753/jis.2021.1778

Keywords:

Web Accessibility, Mobile Accessibility, Evaluation Methods, Systematic Mapping

Abstract

The use of websites and mobile applications has become essential for numerous daily activities. However, not everyone can have full access to such services and content due to many websites and applications being inaccessible to people with disabilities, such as people with vision impairments. In this context, even though developers may demonstrate an effort to create more accessible content, there is limited information about the characteristics of different accessibility assessment methods applied to websites and mobile applications. Thus, the present study aimed to perform a meta-analysis of 38 types of accessibility problems on websites and mobile applications extracted from 38 studies in the literature from an initial search of 304 articles. Studies carried out automated assessments using tools, expert-based inspections and user testing involving disabled people. The results confirm other considerations made in the literature, showing that automated evaluation methods have significant limitations on an adequate coverage of accessibility problems, covering less than 40% of the types of problems found on websites and less than 20% on mobile apps. A significant percentage of problems both on mobile and web platforms were only encountered by studies involving users. Expert inspection showed a higher coverage of problems encountered by users, both on mobile apps and on websites, despite not covering all of them. Thus, the article concludes by showing a consolidation of literature data to reinforce that effective accessibility evaluations of web and mobile applications should count in expert-based inspections and user tests involving people with disabilities.

Downloads

Download data is not yet available.

References

Abdolrahmani, A. and Kuber, R. (2016). Should i trust it when i cannot see it? credibility assessment for blind web users. In Proceedings of the 18th International ACM SIGACCESS Conference on Computers and Accessibility, pages 191–199.

Abou-Zahra, S. (2008). Web accessibility evaluation. In Harper, S. and Yesilada, Y., editors, Web Accessibility: A Foundation for Research, pages 79–106. Springer.

Acosta-Vargas, P., Salvador-Acosta, B., Zalakeviciute, R., Alexandrino, K., Pérez-Medina, J.-L., Rybarczyk, Y., and Gonzalez, M. (2020). Accessibility assessment of mobile meteorological applications for users with low vision. In International Conference on Applied Human Factors and Ergonomics, pages 199–205. Springer.

Acosta-Vargas, P., Salvador-Ullauri, L., Pérez-Medina, J. L., Zalakeviciute, R., and Hernandez, W. (2019a). Heuristic method of evaluating accessibility of mobile in selected applications for air quality monitoring. In International Conference on Applied Human Factors and Ergonomics, pages 485–495. Springer.

Acosta-Vargas, P., Salvador-Ullauri, L. A., and Luján-Mora, S. (2019c). A heuristic method to evaluate web accessibility for users with low vision. IEEE Access, 7:125634–125648.

Agarwal, V., Poddar, S., and Karnavat, S. J. (2020). A studyon growth of mobile banking in india during covid-19. PalArch’s Journal of Archaeology of Egypt / Egyptology, 17:9461–9485.

Aizpurua, A., Arrue, M., Harper, S., and Vigo, M. (2014). Are users the gold standard for accessibility evaluation? In Proceedings of the 11th Web for All Conference, pages 1–4.

Aizpurua, A., Arrue, M., and Vigo, M. (2015). Prejudices, memories, expectations and confidence influence experienced accessibility on the web. Computers in Human Behavior, 51:152–160.

Akram, M. and Bt Sulaiman, R. (2020). An empirical study to evaluate the accessibility of arabic websites by low vision users. In 2020 8th International Conference on Information Technology and Multimedia (ICIMU), pages 206–211.

Archambault, D. et al. (2017). Web widgets barriers for visually impaired users. Studies in Health Technology and Informatics, 242:836–842.

Brajnik, G. (2008). A comparative test of web accessibility evaluation methods. In Proceedings of the 10th international ACM SIGACCESS conference on Computers and accessibility, pages 113–120.

Brajnik, G., Yesilada, Y., and Harper, S. (2011). The expertise effect on web accessibility evaluation methods. Human–Computer Interaction, 26(3):246–283.

Carvalho, L. P., Dias, F. S., and Freire, A. P. (2018a). An analysis of five different native and web-hybrid platforms for building android apps and their accessibility for screen readers. SBC Journal on Interactive Systems, 9(3):20–33.

Carvalho, L. P. and Freire, A. P. (2017). Native or web hybrid apps? an analysis of the adequacy for accessibility of android interface components used with screen readers. In Proceedings of the XVI Brazilian Symposium on Human Factors in Computing Systems, IHC 2017, New York, NY, USA. Association for Computing Machinery.

Carvalho, L. P., Peruzza, B. P. M., Santos, F., Ferreira, L. P., and Freire, A. P. (2016). Accessible smart cities? inspecting the accessibility of brazilian municipalities’ mobile applications. In Proceedings of the 15th Brazilian Symposium on Human Factors in Computing Systems, pages 1–10.

Carvalho, M. C. N., Dias, F. S., Reis, A. G. S., and Freire, A. P. (2018b). Accessibility and usability problems encountered on websites and applications in mobile devices by blind and normal-vision users. In Proceedings of the 33rd Annual ACM Symposium on Applied Computing, SAC ’18, page 2022–2029, New York, NY, USA. Association for Computing Machinery.

da Silva, C. F., Ferreira, S. B. L., and Ramos, J. a. F. M. (2016a). Whatsapp accessibility from the perspective of visually impaired people. In Proceedings of the 15th Brazilian Symposium on Human Factors in Computing Systems, IHC ’16, New York, NY, USA. Association for Computing Machinery.

da Silva, C. F., Ferreira, S. B. L., and Ramos, J. F. M. (2016b). Whatsapp accessibility from the perspective of visually impaired people. In Proceedings of the 15th Brazilian Symposium on Human Factors in Computing Systems, pages 1–10.

da Silva, C. F., Ferreira, S. B. L., and Sacramento, C. (2018a). Mobile application accessibility in the context of visually impaired users. In Proceedings of the 17th Brazilian Symposium on Human Factors in Computing Systems, IHC 2018, New York, NY, USA. Association for Computing Machinery.

da Silva, C. F., Ferreira, S. B. L., and Sacramento, C. (2018b). Mobile application accessibility in the context of visually impaired users. In Proceedings of the 17th Brazilian Symposium on Human Factors in Computing Systems, pages 1–10.

de Oliveira, G. A. A., de Bettio, R. W., and Freire, A. P. (2016). Accessibility of the smart home for users with visual disabilities: an evaluation of open source mobile applications for home automation. In Proceedings of the 15th Brazilian Symposium on Human Factors in Computing Systems, pages 1–10.

Doush, I. and AlMeraj, Z. (2019). Evaluating the accessibility of kuwaiti e-government websites. Jordanian Journal of Computers and Information Technology (JJCIT), 5(03).

Eler, M. M., Rojas, J. M., Ge, Y., and Fraser, G. (2018). Automated accessibility testing of mobile apps. In 2018 IEEE 11th International Conference on Software Testing, Verification and Validation (ICST), pages 116–126. IEEE.

Fernandes, N., Guerreiro, T., Marques, D., and Carriço, L. (2015). Optimus web: selective delivery of desktop or mobile web pages. In Proceedings of the 12th Web for All Conference, pages 1–4. for Standardization, I. I. O. (2018). Iso 9241-11. usability: Definitions and concepts. Retrieved April 23, 2020 from https://www.iso.org/obp/ui/iso:std:iso:9241:-11:ed2:v1:en.

Freire, A. P. (2012). Disabled people and the Web: Userbased measurement of accessibility. PhD thesis, University of York.

Gadde, P. and Bolchini, D. (2014). From screen reading to aural glancing: towards instant access to key page sections. In Proceedings of the 16th international ACM SIGACCESS conference on Computers & accessibility, pages 67–74.

Galkute, M. et al. (2020). Improving the web accessibility of a university library for people with visual disabilities through a mixed evaluation approach. In International Conference on Human-Computer Interaction, pages 56–71. Springer.

Ghidini, E., Almeida, W. D. L., Manssour, I. H., and Silveira, M. S. (2016). Developing apps for visually impaired people: Lessons learned from practice. In 2016 49th Hawaii International Conference on System Sciences (HICSS), pages 5691–5700.

Gonçalves, R., Rocha, T., Martins, J., Branco, F., and AuYong-Oliveira, M. (2018). Evaluation of e-commerce websites accessibility and usability: an e-commerce platform analysis with the inclusion of blind users. Universal Access in the Information Society, 17(3):567–583.

Governo Brasileiro (2014). emag - modelo de acessibilidadeem governo eletrônico. Retrieved September 11, 2020 from http://emag.governoeletronico.gov.br/

Hanson, V. L. and Richards, J. T. (2013). Progress on website accessibility? ACM Trans. Web, 7(1).

Harrison, C. and Petrie, H. (2007). Severity of usability and accessibility problems in ecommerce and government websites. In People and Computers XX—Engage, pages 255–262. Springer.

Ismail, A., Kuppusamy, K., and Nengroo, A. S. (2018). Multitool accessibility assessment of government department websites: a case-study with jkgad. Disability and Rehabilitation: Assistive Technology, 13(6):504–516.

ISO (2018). Iso 9241-11. usability: Definitionsand concepts. International Organization for Standardization. Retrieved April 23, 2020 from https://www.iso.org/obp/ui/iso:std:iso:9241:-11:ed-2:v1:en

Ivory, M. Y. (2013). Automated Web Site Evaluation: Researchers’ and Practioners’ Perspectives, volume 4. Springer Science & Business Media.

Jaeger, P. T. (2006). Assessing section 508 compliance on federal e-government web sites: A multi-method, user-centered evaluation of accessibility for persons with disabilities. Government Information Quarterly, 23(2):169–190.

Kim, H. K., Han, S. H., Park, J., and Park, J. (2016). The interaction experiences of visually impaired people with assistive technology: A case study of smartphones. International Journal of Industrial Ergonomics, 55:22–33.

Kulpa, C. C. and Amaral, F. G. (2014). Evaluation of tablet pc application interfaces with low vision users: Focusing on usability. In International Conference of Design, User Experience, and Usability, pages 273–284. Springer.

Lazar, J. (2005). Web usability: A user-centered design approach. Addison-Wesley Longman Publishing Co., Inc.

Loureiro, J. R., Cagnin, M. I., and Paiva, D. M. (2014). Web accessibility in social networking services. In International Conference on Computational Science and Its Applications, pages 586–601. Springer.

Loureiro, J. R., Cagnin, M. I., and Paiva, D. M. B. (2015). Analysis of web accessibility in social networking services through blind users’ perspective and an accessible prototype. In International Conference on Computational Science and Its Applications, pages 117–131. Springer.

Mateus, D. A., Silva, C. A., Eler, M. M., and Freire, A. P. (2020). Accessibility of mobile applications: Evaluation by users with visual impairment and by automated tools. In Proceedings of the 19th Brazilian Symposium on Human Factors in Computing Systems, IHC ’20, New York, NY, USA. Association for Computing Machinery.

Mátrai, R. K. (2018). How to make an electronic library accessible. The electronic library.

Mounika, P., Karia, D., Sharma, K., and Biswas, P. (2019). Accessibility evaluation of three important indian websites. In Research into Design for a Connected World, pages 243–254. Springer.

Nogueira, T. C., Ferreira, D. J., Carvalho, S. T., and Berreta, L. O. (2017). Evaluating responsive web design’s impact on blind users. IEEE MultiMedia, 24(2):86–95.

Park, K., So, H.-J., and Cha, H. (2019). Digital equity and accessible moocs: Accessibility evaluations of mobile moocs for learners with visual impairments. Australasian Journal of Educational Technology, 35(6):48–63.

Pereira, L. S. and Archambault, D. (2018). Correlating navigation barriers on web 2.0 with accessibility guidelines. In International Conference on Computers Helping People with Special Needs, pages 13–21. Springer.

Petrie, H. and Bevan, N. (2009). The evaluation of accessibility, usability, and user experience. In Stephanidis, C., editor, The Universal Access Handbook, pages 1–16. CRC Press.

Power, C., Freire, A., Petrie, H., and Swallow, D. (2012). Guidelines are only half of the story: accessibility problems encountered by blind users on the web. In Proceedings of the SIGCHI conference on human factors in computing systems, pages 433–442.

Power, C., Petrie, H., Freire, A. P., and Swallow, D. (2011).Remote evaluation of wcag 2.0 techniques by web users with visual disabilities. In International Conference on Universal Access in Human-Computer Interaction, pages 285–294. Springer.

Power, J. A. (2018). Ebsco information services usability study on accessibility. Reference Services Review.

Pribeanu, C., Fogarassy-Neszly, P., and Pătru, A. (2014). Municipal web sites accessibility and usability for blind users: preliminary results from a pilot study. Universal access in the information society, 13(3):339–349.

Quispe, F. E. M. and Eler, M. M. (2018). Accessibility recommendations for mobile applications: a contribution to the brazilian digital government standards. In Proceedings of the XIV Brazilian Symposium on Information Systems, pages 1–8.

Rodrigues, L. A. and Prietch, S. S. (2018). Analysis, redesign and validation of accessibility resources applied to an official electronic journal for the promotion of equal access to public acts. In Proceedings of the 17th Brazilian Symposium on Human Factors in Computing Systems, pages 1–10.

Rømen, D. and Svanæs, D. (2012). Validating wcag versions 1.0 and 2.0 through usability testing with disabled users. Universal Access in the Information Society, 11(4):375–385.

Rysavy, M. D. and Michalak, R. (2020). Assessing the accessibility of library tools & services when you aren’t an accessibility expert: Part 1. Journal of Library Administration, 60(1):71–79.

Silva, C. A., de Oliveira, A. F., Mateus, D. A., Costa, H. A. X., and Freire, A. P. (2019). Types of problems encountered by automated tool accessibility assessments, expert inspections and user testing: a systematic literature mapping. In Proceedings of the 18th Brazilian Symposium on Human Factors in Computing Systems, pages 1–11.

Stephanidis, C. (2009). The universal access handbook. CRC Press.

Valencia, X., Arrue, M., Rojas-Valduciel, H., and Moreno, L. (2014). Interdependent components for the development of accessible xul applications for screen reader users. In WEBIST (2), pages 65–73.

Vigo, M., Brown, J., and Conway, V. (2013). Benchmarking web accessibility evaluation tools: Measuring the harm of sole reliance on automated tests. In Proceedings of the 10th International Cross-Disciplinary Conference on Web Accessibility, W4A ’13, New York, NY, USA. Association for Computing Machinery.

W3C (2016). Web accessibility evaluation tools list. Retrieved December 15, 2020 from https://www.w3.org/WAI/ER/tools/.

W3C (2019). http://www.w3c.br/pub/materiais/publicacoesw3c/cartilha-w3cbr-acessibilidade-webfasciculo-i.html. Retrieved April 15, 2019 from https://www.w3.org/WAI/ER/tools/.

W3C (2020). https://www.w3.org/wai/standardsguidelines/wcag/. Retrieved April 15, 2019 from https://www.w3.org/WAI/ER/tools/.

Wentz, B., Pham, D., Feaser, E., Smith, D., Smith, J., and Wilson, A. (2019). Documenting the accessibility of 100 us bank and finance websites. Universal Access in the Information Society, 18(4):871–880.

Yan, S. and Ramachandran, P. (2019). The current status of accessibility in mobile apps. ACM Transactions on Accessible Computing (TACCESS), 12(1):1–31.

Yi, Y. J. (2020). Web accessibility of healthcare web sites of korean government and public agencies: a user test for persons with visual impairment. Universal Access in the Information Society, 19(1):41–56.

Zaphiris, P. (2007). Web usability: A user-centered design approach by jonathan lazar (2006): Isbn 0321321359, publisher: Addison-wesley. Univers. Access Inf. Soc., 6(1):115.

Zitkus, E., Brigatto, A. C., Ferrari, A. L. M., Bonfim, G. H., Carvalho Filho, I. F., Reis, T. D., Medola, F. O., and Paschoarelli, L. C. (2016). Accessibility and usability of websites intended for people with disabilities: a preliminary study. In International Conference of Design, User Experience, and Usability, pages 678–688. Springer.