Analysis of Performance and Fairness of Classification Algorithms with Class and Protected Attribute Noise

Ana Vargas; Rosa Delgadillo

doi:10.5753/jbcs.2026.5960

Authors

Ana Vargas National University of San Marcos https://orcid.org/0000-0002-1308-574X
Rosa Delgadillo National University of San Marcos https://orcid.org/0000-0003-4899-3008

DOI:

https://doi.org/10.5753/jbcs.2026.5960

Keywords:

Classification, Noise model, Fairness, Accuracy

Abstract

Noise in data is an underexplored source of bias. This study investigates the impact of noise on both the performance and fairness of three classic classifiers: Random Forest (RF), Support Vector Machine (SVM), and Logistic Regression (RL), using four public datasets for binary classification. A comparative analysis is conducted to examine the effects of different types and levels of noise introduced in a predictor variable (protected attribute) and/or the response variable (class). Performance is measured using accuracy (ACC), while model fairness is evaluated using the Average Absolute Odds Difference (AAOD) metric, with a binary attribute as the protected. We results suggest that injecting noise into both the class label and the protected attribute (cn) yields results similar to injecting noise only into the labels (ln). The observed robustness of the Average Absolute Odds Difference (AAOD) under these conditions warrants careful interpretation. As AAOD values tend to decrease or remain stable despite increasing noise levels, there is a risk that the metric may be insensitive to noise. This suggests that AAOD might artificially report higher fairness in scenarios where significant underlying disparities actually persist.

Downloads

Download data is not yet available.

References

Alharbi, A. A. (2024). Classification performance analysis of decision tree-based algorithms with noisy class variable. Discrete Dynamics in Nature and Society, 2024(1):6671395. DOI: 10.1155/2024/6671395.

Angwin, J., Larson, J., Mattu, S., and Kirchner, L. (2016). Machine bias: There’s software used across the country to predict future criminals and it’s biased against blacks. ProPublica. Available at:[link].

Barocas, S., Hardt, M., and Narayanan, A. (2017). Fairness in machine learning. NIPS Tutorial. Available at:[link].

Becker, B. and Kohavi, R. (1996). Adult. UCI Machine Learning Repository. DOI: 10.24432/C5XW20.

Blanzeisky, W. and Cunningham, P. (2022). Using pareto simulated annealing to address algorithmic bias in machine learning. The Knowledge Engineering Review, 37:e5. DOI: 10.1017/S0269888922000029.

Canalli, Y., Braida, F., Alvim, L., and Zimbrão, G. (2024). Fair transition loss: From label noise robustness to bias mitigation. Knowledge-Based Systems, 294:111711. DOI: 10.1016/j.knosys.2024.111711.

Caton, S. and Haas, C. (2024). Fairness in machine learning: A survey. ACM Computing Surveys, 56(7):1–38. DOI: 10.1145/3616865.

Celis, L. E., Huang, L., Keswani, V., and Vishnoi, N. K. (2021). Fair classification with noisy protected attributes: A framework with provable guarantees. In Meila, M. and Zhang, T., editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 1349-1361. PMLR. Available at:[link].

Chouldechova, A. (2017). Fair prediction with disparate impact: A study of bias in recidivism prediction instruments. Big data, 5(2):153-163. DOI: 10.1089/big.2016.0047.

Corbett-Davies, S., Pierson, E., Feller, A., Goel, S., and Huq, A. (2017). Algorithmic decision making and the cost of fairness. In Proceedings of the 23rd acm sigkdd international conference on knowledge discovery and data mining, pages 797-806. DOI: 10.1145/3097983.3098095.

Davani, A. M., Díaz, M., and Prabhakaran, V. (2022). Dealing with disagreements: Looking beyond the majority vote in subjective annotations. Transactions of the Association for Computational Linguistics, 10:92-110. DOI: 10.1162/tacl_a_00449.

Dwork, C., Hardt, M., Pitassi, T., Reingold, O., and Zemel, R. (2012). Fairness through awareness. In Proceedings of the 3rd innovations in theoretical computer science conference, pages 214-226. DOI: 10.1145/2090236.2090255.

Ferry, J., Aivodji, U., Gambs, S., Huguet, M.-J., and Siala, M. (2023). Improving fairness generalization through a sample-robust optimization method. Machine Learning, 112(6):2131-2192. DOI: 10.1007/s10994-022-06191-y.

Friedler, S. A., Scheidegger, C., Venkatasubramanian, S., Choudhary, S., Hamilton, E. P., and Roth, D. (2019). A comparative study of fairness-enhancing interventions in machine learning. In Proceedings of the conference on fairness, accountability, and transparency, pages 329-338. DOI: 10.1145/3287560.3287589.

Gao, X., Zhai, J., Ma, S., Shen, C., Chen, Y., and Wang, Q. (2022). Fairneuron: improving deep neural network fairness with adversary games on selective neurons. In Proceedings of the 44th International Conference on Software Engineering, ICSE '22, page 921–933, New York, NY, USA. Association for Computing Machinery. DOI: 10.1145/3510003.3510087.

Ghosh, A., Dutt, R., and Wilson, C. (2021). When fair ranking meets uncertain inference. In Proceedings of the 44th international ACM SIGIR conference on research and development in information retrieval, pages 1033-1043. DOI: 10.1145/3404835.3462850.

Ghosh, A., Jagielski, M., and Wilson, C. (2022). Subverting fair image search with generative adversarial perturbations. In Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency, pages 637-650. DOI: 10.1145/3531146.3533128.

Ghosh, A., Kvitca, P., and Wilson, C. (2023). When fair classification meets noisy protected attributes. In Proceedings of the 2023 AAAI/ACM Conference on AI, Ethics, and Society, pages 679-690. DOI: 10.1145/3600211.3604707.

Grazzi, R., Akhavan, A., Falk, J. I., Cella, L., and Pontil, M. (2022). Group meritocratic fairness in linear contextual bandits. In Koyejo, S., Mohamed, S., Agarwal, A., Belgrave, D., Cho, K., and Oh, A., editors, Advances in Neural Information Processing Systems, volume 35, pages 24392-24404. Curran Associates, Inc.. DOI: 10.52202/068431-1771.

Gupta, S. and Gupta, A. (2019). Dealing with noise problem in machine learning data-sets: A systematic review. Procedia Computer Science, 161:466-474. DOI: 10.1016/j.procs.2019.11.146.

Hardt, M., Price, E., and Srebro, N. (2016). Equality of opportunity in supervised learning. Advances in neural information processing systems, 29. DOI: 10.48550/arXiv.1610.02413.

Hasan, R. and Chu, C. (2022). Noise in datasets: What are the impacts on classification performance?[noise in datasets: What are the impacts on classification performance?]. In Proceedings of the 11th International Conference on Pattern Recognition Applications and Methods. DOI: 10.5220/0010782200003122.

Hofmann, H. (1994). Statlog (German Credit Data). UCI Machine Learning Repository. DOI: 10.24432/C5NC77.

Hort, M., Chen, Z., Zhang, J. M., Harman, M., and Sarro, F. (2024). Bias mitigation for machine learning classifiers: A comprehensive survey. ACM Journal on Responsible Computing, 1(2):1-52. DOI: 10.1145/3631326.

Hort, M., Zhang, J. M., Sarro, F., and Harman, M. (2021). Fairea: A model behaviour mutation approach to benchmarking bias mitigation methods. In Proceedings of the 29th ACM joint meeting on European software engineering conference and symposium on the foundations of software engineering, pages 994-1006. DOI: 10.1145/3468264.3468565.

Hossain, S., Micha, E., and Shah, N. (2021). Fair algorithms for multi-agent multi-armed bandits. In Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P., and Vaughan, J. W., editors, Advances in Neural Information Processing Systems, volume 34, pages 24005-24017. Curran Associates, Inc.. DOI: 10.48550/arXiv.2007.06699.

Huang, W., Labille, K., Wu, X., Lee, D., and Heffernan, N. (2022). Achieving user-side fairness in contextual bandits. Human-Centric Intelligent Systems, 2(3):81-94. DOI: 10.1007/s44230-022-00008-w.

Khani, F. (2021). Causes, Measurement, and Mitigation of Loss Discrepancy. Stanford University. Available at:[link].

Khani, F. and Liang, P. (2020). Feature noise induces loss discrepancy across groups. In III, H. D. and Singh, A., editors, Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 5209-5219. PMLR. Available at:[link].

Kuzilek, Jakub, H. M. and Zdrahal, Z. (2015). Open University Learning Analytics dataset. UCI Machine Learning Repository. DOI: 10.24432/C5KK69.

Le Quy, T., Roy, A., Iosifidis, V., Zhang, W., and Ntoutsi, E. (2022). A survey on datasets for fairness-aware machine learning. Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery, 12(3):e1452. DOI: 10.1002/widm.1452.

Mehrabi, N., Morstatter, F., Saxena, N., Lerman, K., and Galstyan, A. (2021). A survey on bias and fairness in machine learning. ACM computing surveys (CSUR), 54(6):1-35. DOI: 10.1145/3457607.

Patil, V., Ghalme, G., Nair, V., and Narahari, Y. (2021). Achieving fairness in the stochastic multi-armed bandit problem. Journal of Machine Learning Research, 22(174):1-31. Available at:[link].

Pessach, D. and Shmueli, E. (2022). A review on fairness in machine learning. ACM Computing Surveys (CSUR), 55(3):1-44. DOI: 10.1145/3494672.

Roh, Y., Lee, K., Whang, S. E., and Suh, C. (2021). Fairbatch: Batch selection for model fairness. DOI: 10.48550/arXiv.2012.01696.

Sáez, J. A. (2022). Noise models in classification: Unified nomenclature, extended taxonomy and pragmatic categorization. Mathematics, 10(20):3736. DOI: 10.3390/math10203736.

Silva, I. O. e., Soares, C., Sousa, I., and Ghani, R. (2024). Systematic analysis of the impact of label noise correction on ml fairness. In Liu, T., Webb, G., Yue, L., and Wang, D., editors, AI 2023: Advances in Artificial Intelligence, pages 173-184, Singapore. Springer Nature Singapore. DOI: 10.1007/978-981-99-8391-9_14.

Stefano, P. G. D., Hickey, J. M., and Vasileiou, V. (2020). Counterfactual fairness: removing direct effects through regularization. CoRR, abs/2002.10774. DOI: 10.48550/arXiv.2002.10774.

Wang, J., Liu, Y., and Levy, C. (2021). Fair classification with group-dependent label noise. In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, FAccT '21, page 526–536, New York, NY, USA. Association for Computing Machinery. DOI: 10.1145/3442188.3445915.

Wu, S., Gong, M., Han, B., Liu, Y., and Liu, T. (2022). Fair classification with instance-dependent label noise. In Schölkopf, B., Uhler, C., and Zhang, K., editors, Proceedings of the First Conference on Causal Learning and Reasoning, volume 177 of Proceedings of Machine Learning Research, pages 927-943. PMLR. Available at:[link].

Xia, X., Liu, T., Han, B., Wang, N., Gong, M., Liu, H., Niu, G., Tao, D., and Sugiyama, M. (2020). Part-dependent label noise: Towards instance-dependent label noise. In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., and Lin, H., editors, Advances in Neural Information Processing Systems, volume 33, pages 7597-7610. Curran Associates, Inc. Available at:[link].

Yeom, S. and Fredrikson, M. (2020). Individual fairness revisited: Transferring techniques from adversarial robustness. CoRR, abs/2002.07738. DOI: 10.24963/ijcai.2020/61.

Zemel, R., Wu, Y., Swersky, K., Pitassi, T., and Dwork, C. (2013). Learning fair representations. In International conference on machine learning, pages 325-333. PMLR. Available at:[link].