A Deep Learning Model for the Assessment of the Visual Aesthetics of Mobile User Interfaces

Adriano Luiz de Souza Lima; Christiane Gresse von Wangenheim; Osvaldo P. H. R. Martins; Aldo von Wangenheim; Jean C. R. Hauck; Adriano Ferreti Borgatto

doi:10.5753/jbcs.2024.3255

Authors

Adriano Luiz de Souza Lima Federal University of Santa Catarina https://orcid.org/0000-0002-5264-5746
Christiane Gresse von Wangenheim Federal University of Santa Catarina https://orcid.org/0000-0002-6566-1606
Osvaldo P. H. R. Martins Federal University of Santa Catarina https://orcid.org/0000-0002-3878-8417
Aldo von Wangenheim Federal University of Santa Catarina https://orcid.org/0000-0003-4532-1417
Jean C. R. Hauck Federal University of Santa Catarina https://orcid.org/0000-0001-6550-9092
Adriano Ferreti Borgatto Federal University of Santa Catarina https://orcid.org/0000-0001-6280-2525

DOI:

https://doi.org/10.5753/jbcs.2024.3255

Keywords:

Aesthetics, Mobile Application, Android, Deep Learning, Automatic Assessment

Abstract

Visual aesthetics is one of the first aspects that users experience when looking at graphical user interfaces (GUIs), contributing to the perceived usability and credibility of a software system. It can also be an essential success factor in contexts where graphical elements play an important role in attracting users, such as choosing a mobile app from an app store. Therefore, visual aesthetics assessments are crucial in interface design, but traditional methods, involving target user representatives assessing each GUI individually, are costly and time-consuming. In this context, machine learning models have been demonstrated to be promising in automating the assessment of GUIs of web-based software systems. Yet, solutions for the assessment of mobile GUIs using machine learning are still unknown. Here we introduce a deep learning model to assess the visual aesthetics of mobile Android applications designed with App Inventor. We used a supervised learning approach to train and compare models using three different architectures. The highest performing model, a Resnet50, achieved a mean squared error of .022. The assessments of new GUIs showed an excellent correlation with human ratings (ρ = .9), and the Bland Altman plot analysis revealed 95% agreement with their labels. These results indicate the model’s effectiveness in automating the visual aesthetics assessment of GUIs of mobile apps.

Downloads

Download data is not yet available.

References

Aggarwal, C. C. (2018). Neural Networks and Deep Learning: A Textbook. Springer International Publishing, Cham. DOI: 10.1007/978-3-319-94463-0.

Alemerien, K. and Magel, K. (2014). GUIEvaluator: A Metric-tool for Evaluating the Complexity of Graphical User Interfaces. In Proceedings of the Twenty-Sixth International Conference on Software Engineering & Knowledge Engineering, pages 13-18, Vancouver, BC, Canada. Available online [link].

Alves, N. d. C., Gresse Von Wangenheim, C., and Hauck, J. C. R. (2019). Approaches to Assess Computational Thinking Competences Based on Code Analysis in K-12 Education: A Systematic Mapping Study. Informatics in Education, 18(1):17-39. Available online [link].

Amershi, S., Begel, A., Bird, C., DeLine, R., Gall, H., Kamar, E., Nagappan, N., Nushi, B., and Zimmermann, T. (2019). Software Engineering for Machine Learning: A Case Study. In 2019 IEEE/ACM 41st International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP), pages 291-300. DOI: 10.1109/ICSE-SEIP.2019.00042.

Anderson, S. P. (2011). Seductive Interaction Design: Creating Playful, Fun, and Effective User Experiences. New Riders Pub, Berkeley, CA, 1st edition edition. Book.

Ashmore, R., Calinescu, R., and Paterson, C. (2021). Assuring the Machine Learning Lifecycle: Desiderata, Methods, and Challenges. ACM Computing Surveys, 54(5):111:1-111:39. DOI: 10.1145/3453444.

Bakaev, M., Heil, S., Chirkov, L., and Gaedke, M. (2022). Benchmarking Neural Networks-Based Approaches for Predicting Visual Perception of User Interfaces. In Degen, H. and Ntoa, S., editors, Artificial Intelligence in HCI, Lecture Notes in Computer Science, pages 217-231, Cham. Springer International Publishing. DOI: 10.1007/978-3-031-05643-7_14.

Bhandari, U., Chang, K., and Neben, T. (2019). Understanding the Impact of Perceived Visual Aesthetics on User Evaluations: An Emotional perspective. Information & Management, 56(1):85-93. DOI: 10.1016/j.im.2018.07.003.

Bhandari, U., Neben, T., Chang, K., and Chua, W. Y. (2017). Effects of Interface Design Factors on Affective Responses and Quality Evaluations in Mobile Applications. Computers in Human Behavior, 72:525-534. Place: Netherlands Publisher: Elsevier Science. DOI: 10.1016/j.chb.2017.02.044.

Bland, J. M. and Altman, D. G. (1986). Statistical Methods for Assessing Agreement Between Two Methods of Clinical Measurement. The Lancet, 327(8476):307-310. DOI: 10.1016/S0140-6736(86)90837-8.

Bonett, D. G. and Wright, T. A. (2000). Sample size requirements for estimating pearson, kendall and spearman correlations. Psychometrika, 65(1):23-28. DOI: 10.1007/BF02294183.

Botchkarev, A. (2019). A New Typology Design of Performance Metrics to Measure Errors in Machine Learning Regression Algorithms. Interdisciplinary Journal of Information, Knowledge, and Management, 14:045-076. DOI: 10.28945/4184.

Bryman, A. and Cramer, D. (1990). Quantitative Data Analysis for Social Scientists. Quantitative data analysis for social scientists. Taylor & Francis/Routledge, Florence, KY, US. Pages: xiv, 290.

Choi, J. H. and Lee, H.-J. (2012). Facets of Simplicity for the Smartphone Interface: A Structural Model. International Journal of Human-Computer Studies, 70(2):129-142. DOI: 10.1016/j.ijhcs.2011.09.002.

Deng, Y., Loy, C. C., and Tang, X. (2017). Image Aesthetic Assessment: An experimental survey. IEEE Signal Processing Magazine, 34(4):80-106. Conference Name: IEEE Signal Processing Magazine. DOI: 10.1109/MSP.2017.2696576.

Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., and Houlsby, N. (2021). An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. arXiv:2010.11929 [cs]. arXiv: 2010.11929. DOI: 10.48550/arXiv.2010.11929.

Dou, Q., Zheng, X. S., Sun, T., and Heng, P.-A. (2019). Webthetics: Quantifying Webpage Aesthetics with Deep Learning. International Journal of Human-Computer Studies, 124:56-66. DOI: 10.1016/j.ijhcs.2018.11.006.

Flora, H. K., Wang, X., and V.Chande, S. (2014). An Investigation into Mobile Application Development Processes: Challenges and Best Practices. International Journal of Modern Education and Computer Science (IJMECS), 6(6):1. DOI: 10.5815/ijmecs.2014.06.01.

Fonnegra, R. D., Blair, B., and Díaz, G. M. (2017). Performance Comparison of Deep Learning Frameworks in Image Classification Problems Using Convolutional and Recurrent Networks. In 2017 IEEE Colombian Conference on Communications and Computing (COLCOM), pages 1-6. DOI: 10.1109/ColComCon.2017.8088219.

Giavarina, D. (2015). Understanding Bland Altman Analysis. Biochemia Medica, 25(2):141-151. DOI: 10.11613/BM.2015.015.

Gresse von Wangenheim, C., Hauck, J. C. R., Demetrio, M. F., Pelle, R., Alves, N. d. C., Azevedo, L. F., and Barbosa, H. (2018a). CodeMaster - Automatic Assessment and Grading of App Inventor and Snap! Programs. Informatics in Education - An International Journal, 17(1):117-150. Available online [link].

Gresse von Wangenheim, C., Porto, J. V. A., Hauck, J. C. R., and Borgatto, A. F. (2018b). Do We Agree on User Interface Aesthetics of Android Apps? arXiv, pages 1-5. arXiv: 1812.09049. DOI: 10.48550/arXiv.1812.09049.

Hamborg, K.-C., Hülsmann, J., and Kaspar, K. (2014). The Interplay between Usability and Aesthetics: More Evidence for the “What Is Usable Is Beautiful” Notion. Advances in Human-Computer Interaction, 2014:e946239. Publisher: Hindawi. DOI: 10.1155/2014/946239.

He, K., Zhang, X., Ren, S., and Sun, J. (2016). Deep Residual Learning for Image Recognition. pages 770-778. Available online [link].

Howard, J. and Gugger, S. (2020). Fastai: A Layered API for Deep Learning. Information, 11(2):108. Number: 2 Publisher: Multidisciplinary Digital Publishing Institute. DOI: 10.3390/info11020108.

Huang, K.-Y. (2009). Challenges in Human-Computer Interaction Design for Mobile Devices. In Proceedings of the World Congress on Engineering and Computer Science, volume 1, pages 236-241, San Francisco, USA. Available online [link].

Iman, M., Arabnia, H. R., and Rasheed, K. (2023). A Review of Deep Transfer Learning and Recent Advancements. Technologies, 11(2):40. Number: 2 Publisher: Multidisciplinary Digital Publishing Institute. DOI: 10.3390/technologies11020040.

ISO (2011). ISO/IEC 25010:2011, Systems and Software Engineering — Systems and Software Quality Requirements and Evaluation (SQuaRE) — System and Software Quality Models. Available online [link].

Khani, M. G., Mazinani, M. R., Fayyaz, M., and Hoseini, M. (2016). A Novel Approach for Website Aesthetic Evaluation Based on Convolutional Neural Networks. In Proceedings of the 2016 Second International Conference on Web Research (ICWR), pages 48-53. DOI: 10.1109/ICWR.2016.7498445.

Kim, J., Lee, J., and Choi, D. (2003). Designing Emotionally Evocative Homepages: An Empirical Study of the Quantitative Relations Between Design Factors and Emotional Dimensions. International Journal of Human-Computer Studies, 59(6):899-940. DOI: 10.1016/j.ijhcs.2003.06.002.

Kirchner, J., Heberle, A., and Löwe, W. (2015). Classification vs. Regression - Machine Learning Approaches for Service Recommendation Based on Measured Consumer Experiences. In 2015 IEEE World Congress on Services, pages 278-285. ISSN: 2378-3818. DOI: 10.1109/SERVICES.2015.49.

Lavie, T. and Tractinsky, N. (2004). Assessing Dimensions of Perceived Visual Aesthetics of Web Sites. International Journal of Human-Computer Studies, 60(3):269-298. DOI: 10.1016/j.ijhcs.2003.09.002.

LeCun, Y., Bengio, Y., and Hinton, G. (2015). Deep Learning. Nature, 521(7553):436-444. DOI: 10.1038/nature14539.

Leys, C., Ley, C., Klein, O., Bernard, P., and Licata, L. (2013). Detecting outliers: Do not use standard deviation around the mean, use absolute deviation around the median. Journal of Experimental Social Psychology, 49(4):764-766. DOI: 10.1016/j.jesp.2013.03.013.

Li, X., Zhang, G., Li, K., and Zheng, W. (2016). Deep Learning and Its Parallelization. In Buyya, R., Calheiros, R. N., and Dastjerdi, A. V., editors, Big Data: Principles and Paradigms. Morgan Kaufmann. Google-Books-ID: MfOeCwAAQBAJ. DOI: https://doi.org/10.1016/B978-0-12-805394-2.00004-0.

Lima, A. L. d. S. and Gresse von Wangenheim, C. (2021). Assessing the Visual Esthetics of User Interfaces: A Ten-Year Systematic Mapping. International Journal of Human–Computer Interaction, pages 1-21. DOI: 10.1080/10447318.2021.1926118.

Lima, A. L. d. S., Gresse von Wangenheim, C., and Borgatto, A. F. (2022a). Assessment of Visual Aesthetics through Human Judgments: a Systematic Mapping. In Proceedings of the 21st Brazilian Symposium on Human Factors in Computing Systems, IHC '22, pages 1-14, New York, NY, USA. Association for Computing Machinery. DOI: 10.1145/3554364.3560902.

Lima, A. L. d. S., Gresse von Wangenheim, C., and Borgatto, A. F. (2022b). Comparing Scales for the Assessment of Visual Aesthetics of Mobile GUIs Through Human Judgments. International Journal of Mobile Human Computer Interaction (IJMHCI), 14(1):1-28. Publisher: IGI Global. DOI: 10.4018/IJMHCI.313028.

Lima, A. L. d. S., Martins, O. P. H. R., von Wangenheim, C. G., von Wangenheim, A., Borgatto, A. F., and Hauck, J. C. R. (2022c). Automated Assessment of Visual aesthetics of Android User Interfaces with Deep Learning. In Proceedings of the 21st Brazilian Symposium on Human Factors in Computing Systems, IHC '22, pages 1-11, New York, NY, USA. Association for Computing Machinery. DOI: 10.1145/3554364.3559113.

Lin, R. (2022). Augmenting Image Aesthetic Assessment with Diverse Deep Features. In 2021 4th Artificial Intelligence and Cloud Computing Conference, AICCC '21, pages 30-38, New York, NY, USA. Association for Computing Machinery. DOI: 10.1145/3508259.3508264.

Lu, X., Lin, Z., Jin, H., Yang, J., and Wang, J. Z. (2014). RAPID: Rating Pictorial Aesthetics Using Deep Learning. In Proceedings of the 22nd ACM international conference on Multimedia, MM '14, New York, NY, USA. Association for Computing Machinery. DOI: 10.1145/2647868.2654927.

Malu, G., Bapi, R. S., and Indurkhya, B. (2017). Learning Photography Aesthetics with Deep CNNs. arXiv:1707.03981 [cs]. arXiv: 1707.03981. DOI: 10.48550/arXiv.1707.03981.

Miniukovich, A. and De Angeli, A. (2014a). Quantification of Interface Visual Complexity. In Proceedings of the 2014 International Working Conference on Advanced Visual Interfaces, AVI '14, New York, NY, USA. Association for Computing Machinery. DOI: 10.1145/2598153.2598173.

Miniukovich, A. and De Angeli, A. (2014b). Visual Impressions of Mobile App Interfaces. In Proceedings of the 8th Nordic Conference on Human-Computer Interaction: Fun, Fast, Foundational, NordiCHI '14, pages 31-40, New York, NY, USA. Association for Computing Machinery. DOI: 10.1145/2639189.2641219.

Miniukovich, A. and De Angeli, A. (2015). Computation of Interface Aesthetics. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems, pages 1163-1172. Association for Computing Machinery, New York, NY, USA. DOI: 10.1145/2702123.2702575.

MIT App Inventor (2022). MIT App Inventor textbar Explore MIT App Inventor. Available online [link].

Mitchell, T. M. (1997). Machine Learning. McGraw-Hill Education, 1 edition. Book.

Moshagen, M. and Thielsch, M. (2010). Facets of Visual Aesthetics. International Journal of Human-Computer Studies, 68(10):689-709. DOI: 10.1016/j.ijhcs.2010.05.006.

Moshagen, M. and Thielsch, M. (2013). A Short Version of the Visual Aesthetics of Websites Inventory. Behaviour & Information Technology, 32(12):1305-1311. DOI: 10.1080/0144929X.2012.694910.

Norman, D. (2002). Emotion & Design: Attractive Things Work Better. Interactions, 9(4):36-42. DOI: 10.1145/543434.543435.

Nunnally, J. C. and Bernstein, I. H. (1994). Psychometric Theory. McGraw-Hill, New York, 3rd edition. Book.

Paternò, F. (2013). End User Development: Survey of an Emerging Field for Empowering People. ISRN Software Engineering, 2013:e532659. Publisher: Hindawi. DOI: 10.1155/2013/532659.

Polyzotis, N., Roy, S., Whang, S. E., and Zinkevich, M. (2017). Data Management Challenges in Production Machine Learning. In Proceedings of the 2017 ACM International Conference on Management of Data, SIGMOD '17, pages 1723-1726, New York, NY, USA. Association for Computing Machinery. DOI: 10.1145/3035918.3054782.

Punchoojit, L. and Hongwarittorrn, N. (2017). Usability Studies on Mobile User Interface Design Patterns: A Systematic Literature Review. Advances in Human-Computer Interaction, 2017:e6787504. Publisher: Hindawi. DOI: 10.1155/2017/6787504.

Purchase, H. C., Hamer, J., Jamieson, A., and Ryan, O. (2011). Investigating objective measures of web page aesthetics and usability. In Proceedings of the Twelfth Australasian User Interface Conference - Volume 117, AUIC '11, pages 19-28, AUS. Australian Computer Society, Inc. Available online [link].

Rahmat, H., Zulzalil, H., Ghani, A. A. A., and Kamaruddin, A. (2018). A Comprehensive Usability Model for Evaluating Smartphone Apps. Advanced Science Letters, 24(3):1633-1637. DOI: 10.1166/asl.2018.11125.

Ripley, B. D. (2007). Pattern Recognition and Neural Networks. Cambridge University Press. Book.

Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., Berg, A. C., and Fei-Fei, L. (2015). ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision, 115(3):211-252. DOI: 10.1007/s11263-015-0816-y.

Sakaguchi, D., Takimoto, H., and Kanagawa, A. (2022). Study on relationship between composition and prediction of photo aesthetics using CNN. Cogent Engineering, 9(1):2107472. DOI: 10.1080/23311916.2022.2107472.

Schenkman, B. N. and Jönsson, F. U. (2000). Aesthetics and Preferences of Web Pages. Behaviour & Information Technology, 19(5):367-377. DOI: 10.1080/014492900750000063.

Seckler, M., Opwis, K., and Tuch, A. N. (2015). Linking Objective Design Factors with Subjective Aesthetics: An Experimental Study on How Structure and Color of Websites Affect the Facets of Users’ Visual Aesthetic Perception. Computers in Human Behavior, 49:375-389. DOI: 10.1016/j.chb.2015.02.056.

Simonyan, K. and Zisserman, A. (2015). Very Deep Convolutional Networks for Large-Scale Image Recognition. arXiv:1409.1556 [cs].

Smith, L. N. (2018). A Disciplined Approach to Neural Network Hyper-parameters: Part 1 - Learning Rate, Batch Size, Momentum, and Weight Decay. arXiv:1803.09820 [cs, stat]. DOI: 10.48550/arXiv.1803.09820.

Smith, L. N. and Topin, N. (2019). Super-convergence: Very Fast Training of Neural Networks Using Large Learning Rates. In Proceedings of the Artificial Intelligence and Machine Learning for Multi-Domain Operations Applications, volume 11006, page 1100612. International Society for Optics and Photonics. DOI: 10.1117/12.2520589.

Solecki, I., Porto, J., Alves, N. d. C., Gresse von Wangenheim, C., Hauck, J., and Borgatto, A. F. (2020). Automated Assessment of the Visual Design of Android Apps Developed with App Inventor. In Proceedings of the 51st ACM Technical Symposium on Computer Science Education, SIGCSE '20, pages 51-57, New York, NY, USA. Association for Computing Machinery. DOI: 10.1145/3328778.3366868.

Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., and Rabinovich, A. (2015). Going Deeper With Convolutions. pages 1-9. Available online [link].

Taba, S. E. S., Keivanloo, I., Zou, Y., Ng, J., and Ng, T. (2014). An Exploratory Study on the Relation between User Interface Complexity and the Perceived Quality. In Casteleyn, S., Rossi, G., and Winckler, M., editors, Web Engineering, Lecture Notes in Computer Science, pages 370-379, Cham. Springer International Publishing. DOI: 10.1007/978-3-319-08245-5_22.

Tan, M. and Le, Q. V. (2020). EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks.

Tractinsky, N. (2013). Visual Aesthetics. The Encyclopedia of Human Interaction. Available online [link].

Tractinsky, N., Cokhavi, A., Kirschenbaum, M., and Sharfi, T. (2006). Evaluating the Consistency of Immediate Aesthetic Perceptions of Web Pages. International Journal of Human-Computer Studies, 64(11):1071-1083. DOI: 10.1016/j.ijhcs.2006.06.009.

Tractinsky, N., Katz, A., and Ikar, D. (2000). What is Beautiful is Usable. Interacting with Computers, 13(2):127-145. DOI: 10.1016/S0953-5438(00)00031-X.

Tuch, A. N., Roth, S. P., Hornbæk, K., Opwis, K., and Bargas-Avila, J. A. (2012). Is Beautiful Really Usable? Toward Understanding the Relation Between Usability, Aesthetics, and Affect in HCI. Computers in Human Behavior, 28(5):1596-1607. DOI: 10.1016/j.chb.2012.03.024.

Wolber, D., Abelson, H., and Friedman, M. (2015). Democratizing Computing with App Inventor. GetMobile: Mobile Computing and Communications, 18(4):53-58. DOI: 10.1145/2721914.2721935.

Xing, B., Cao, H., Shi, L., Si, H., and Zhao, L. (2022). AI-driven user aesthetics preference prediction for UI layouts via deep convolutional neural networks. Cognitive Computation and Systems, 4(3):250-264. DOI: 10.1049/ccs2.12055.

Xing, B., Si, H., Chen, J., Ye, M., and Shi, L. (2021). Computational model for predicting user aesthetic preference for GUI using DCNNs. CCF Transactions on Pervasive Computing and Interaction, 3(2):147-169. DOI: 10.1007/s42486-021-00064-4.

Zen, M. and Vanderdonckt, J. (2016). Assessing User Interface Aesthetics based on the Inter-subjectivity of Judgment. In Proceedings of the 30th International BCS Human Computer Interaction Conference, Poole, UK. BCS Learning & Development. DOI: 10.14236/ewic/HCI2016.25.

Zhuang, F., Qi, Z., Duan, K., Xi, D., Zhu, Y., Zhu, H., Xiong, H., and He, Q. (2021). A Comprehensive Survey on Transfer Learning. Proceedings of the IEEE, 109(1):43-76. Conference Name: Proceedings of the IEEE. DOI: 10.1109/JPROC.2020.3004555.