Online Deep Learning Hyperparameter Tuning based on Provenance Analysis
DOI:
https://doi.org/10.5753/jidm.2021.1924Keywords:
Deep Learning, Provenance, Hyperparameter tuningAbstract
Training Deep Learning (DL) models require adjusting a series of hyperparameters. Although there are several tools to automatically choose the best hyperparameter configuration, the user is still the main actor to take the final decision. To decide whether the training should continue or try different configurations, the user needs to analyze online the hyperparameters most adequate to the training dataset, observing metrics such as accuracy and loss values. Provenance naturally represents data derivation relationships (i.e., transformations, parameter values, etc.), which provide important support in this data analysis. Most of the existing provenance solutions define their own and proprietary data representations to support DL users in choosing the best hyperparameter configuration, which makes data analysis and interoperability difficult. We present Keras-Prov and its extension, named Keras-Prov++, which provides an analytical dashboard to support online hyperparameter fine-tuning. Different from the current mainstream solutions, Keras-Prov automatically captures the provenance data of DL applications using the W3C PROV recommendation, allowing for hyperparameter online analysis to help the user deciding on changing hyperparameters’ values after observing the performance of the models on a validation set. We provide an experimental evaluation of Keras-Prov++ using AlexNet and a real case study, named DenseED, that acts as a surrogate model for solving equations. During the online analysis, the users identify scenarios that suggest reducing the number of epochs to avoid unnecessary executions and fine-tuning the learning rate to improve the model accuracy.
Downloads
References
Agrawal, T. and Urolagin, S. 2-way arabic sign language translator using CNNLSTM architecture and NLP. In BDET 2020: 2nd International Conference on Big Data Engineering and Technology, Singapore, January 3-5, 2020. ACM, pp. 96–101, 2020.
Badan, F. and Sekanina, L. Optimizing convolutional neural networks for embedded systems by means of neuroevolution. In TPNC 2019. Vol. 11934. pp. 109–121, 2019.
Bengio, Y. Practical recommendations for gradient-based training of deep architectures. In Neural networks: Tricks of the trade. Springer, pp. 437–478, 2012.
Biewald, L. Experiment tracking with weights and biases, 2020. Software available from wandb.com.
Chatzimparmpas, A., Martins, R. M., Kucher, K., and Kerren, A. Visevol: Visual analytics to support hyperparameter search through evolutionary optimization. CoRR vol. abs/2012.01205, 2020.
Chevalier-Boisvert, M., Bahdanau, D., Lahlou, S., Willems, L., Saharia, C., Nguyen, T. H., and Bengio, Y. Babyai: First steps towards grounded language learning with a human in the loop. In International Conference on Learning Representations, 2019.
de Oliveira, G. B., Padilha, R., Dorte, A., Cereda, L., Miyazaki, L., Lopes, M., and Dias, Z. COVID-19 x-ray image diagnostic with deep neural networks. In Advances in Bioinformatics and Computational Biology - 13th Brazilian Symposium on Bioinformatics, BSB 2020, São Paulo, Brazil, November 23-27, 2020, Proceedings, J. C. Setubal and W. M. C. Silva (Eds.). Lecture Notes in Computer Science, vol. 12558. Springer, pp. 57–68, 2020.
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., and Fei-Fei, L. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition. Ieee, pp. 248–255, 2009.
Dozat, T. Incorporating nesterov momentum into adam, 2016.
Duchi, J., Hazan, E., and Singer, Y. Adaptive subgradient methods for online learning and stochastic optimization. Journal of machine learning research 12 (7), 2011.
Fekry, A., Carata, L., Pasquier, T., Rice, A., and Hopper, A. To tune or not to tune? in search of optimal configurations for data analytics. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. New York, NY, USA, pp. 2494–2504, 2020.
Freitas, R. S., Barbosa, C. H., Guerra, G. M., Coutinho, A. L., and Rochinha, F. A. An encoder-decoder deep surrogate for reverse time migration in seismic imaging under uncertainty. arXiv preprint arXiv:2006.09550, 2020.
Freitas, R. S., Barbosa, C. H., Guerra, G. M., Coutinho, A. L., and Rochinha, F. A. An encoder-decoder deep surrogate for reverse time migration in seismic imaging under uncertainty. Computational Geosciences 25 (3): 1229–1250, 2021.
Godoy, W. F., Podhorszki, N., Wang, R., Atkins, C., Eisenhauer, G., Gu, J., Davis, P. E., Choi, J., Germaschewski, K., Huck, K. A., Huebl, A., Kim, M., Kress, J., Kurç, T. M., Liu, Q., Logan, J., Mehta, K., Ostrouchov, G., Parashar, M., Poeschel, F., Pugmire, D., Suchyta, E., Takahashi, K., Thompson, N., Tsutsumi, S., Wan, L., Wolf, M., Wu, K., and Klasky, S. ADIOS 2: The adaptable input output system. A framework for high-performance data management. SoftwareX vol. 12, pp. 100561, 2020.
Goulart, Í., Paes, A., and Clua, E. Learning how to play bomberman with deep reinforcement and imitation learning. In Entertainment Computing and Serious Games - First IFIP TC 14 Joint International Conference, ICEC-JCSG 2019, Arequipa, Peru, November 11-15, 2019, Proceedings, E. D. V. der Spek, S. Göbel, E. Y. Do, E. Clua, and J. B. Hauge (Eds.). Lecture Notes in Computer Science, vol. 11863. Springer, pp. 121–133, 2019.
Guérin, J., Thiery, S., Nyiri, E., Gibaru, O., and Boots, B. Combining pretrained CNN feature extractors to enhance clustering of complex natural images. Neurocomputing vol. 423, pp. 551–571, 2021.
Hoos, H. and Leyton-Brown, K. An efficient approach for assessing hyperparameter importance. In International conference on machine learning. pp. 754–762, 2014.
Ioffe, S. and Szegedy, C. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning. PMLR, pp. 448–456, 2015.
Kingma, D. P. and Ba, J. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 , 2014.
Krizhevsky, A., Sutskever, I., and Hinton, G. E. Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems vol. 25, pp. 1097–1105, 2012.
Krizhevsky, A., Sutskever, I., and Hinton, G. E. Imagenet classification with deep convolutional neural networks. Commun. ACM 60 (6): 84–90, May, 2017.
LeCun, Y., Bengio, Y., and Hinton, G. Deep learning. nature 521 (7553): 436, 2015.
Li, G., Lee, C. H., Jung, J. J., Youn, Y. C., and Camacho, D. Deep learning for EEG data analytics: A survey. Concurr. Comput. Pract. Exp. 32 (18), 2020.
Liashchynskyi, P. and Liashchynskyi, P. Grid search, random search, genetic algorithm: A big comparison for NAS. CoRR vol. abs/1912.06059, 2019.
Liu, Z., Yang, C., Huang, J., Liu, S., Zhuo, Y., and Lu, X. Deep learning framework based on integration of s-mask R-CNN and inception-v3 for ultrasound image-aided diagnosis of prostate cancer. Future Gener. Comput. Syst. vol. 114, pp. 358–367, 2021.
Matsugu, M., Mori, K., Mitari, Y., and Kaneda, Y. Subject independent facial expression recognition with robust face detection using a convolutional neural network. Neural Networks 16 (5): 555–559, 2003. Advances in Neural Networks Research: IJCNN ’03.
McPhillips, T. M., Song, T., Kolisnik, T., Aulenbach, S., Belhajjame, K., Bocinsky, K., Cao, Y., Chirigati, F., Dey, S. C., Freire, J., Huntzinger, D. N., Jones, C., Koop, D., Missier, P., Schildhauer, M., Schwalm, C. R., Wei, Y., Cheney, J., Bieda, M., and Ludäscher, B. Yesworkflow: A user-oriented, language-independent tool for recovering workflow information from scripts. CoRR vol. abs/1502.02403, 2015.
Miao, H., Li, A., Davis, L. S., and Deshpande, A. Modelhub: Lifecycle management for deep learning. Univ. of Maryland, 2015.
Miao, H., Li, A., Davis, L. S., and Deshpande, A. Towards unified data and lifecycle management for deep learning. In 2017 IEEE 33rd ICDE. IEEE, pp. 571–582, 2017.
Moreau, L. and Groth, P. Provenance: an introduction to prov. Synthesis Lectures on the Semantic Web: Theory and Technology 3 (4): 1–129, 2013.
Nilsback, M.-E. and Zisserman, A. A visual vocabulary for flower classification. In 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’06). Vol. 2. IEEE, pp. 1447–1454, 2006.
Özbayoglu, A. M., Gudelek, M. U., and Sezer, O. B. Deep learning for financial applications : A survey. Appl. Soft Comput. vol. 93, pp. 106384, 2020.
Pimentel, J. F., Dey, S. C., McPhillips, T. M., Belhajjame, K., Koop, D., Murta, L., Braganholo, V., and Ludäscher, B. Yin & yang: Demonstrating complementary provenance from noworkflow & yesworkflow. In Provenance and Annotation of Data and Processes - 6th International Provenance and Annotation Workshop, IPAW 2016, McLean, VA, USA, June 7-8, 2016, Proceedings, M. Mattoso and B. Glavic (Eds.). Lecture Notes in Computer Science, vol. 9672. Springer, pp. 161–165, 2016.
Pimentel, J. F., Murta, L., Braganholo, V., and Freire, J. noworkflow: a tool for collecting, analyzing, and managing provenance from python scripts. Proc. VLDB Endow. 10 (12): 1841–1844, 2017.
Pina, D., Kunstmann, L., De Oliveira, D., Valduriez, P., and Mattoso, M. Provenance supporting hyperparameter analysis in deep neural networks. In International Provenance and Annotation Workshop, IPAW, 2021.
Pina, D., Kunstmann, L., Oliveira, D., Valduriez, P., and Mattoso, M. Uma abordagem para coleta e análise de dados de configurações em redes neurais profundas. In Anais do XXXV Simpósio Brasileiro de Bancos de Dados. SBC, Porto Alegre, RS, Brasil, pp. 187–192, 2020.
Pina, D., Neves, L., Paes, A., de Oliveira, D., and Mattoso, M. Análise de hiperparâmetros em aplicações de aprendizado profundo por meio de dados de proveniência. In Anais do XXXIV Simpósio Brasileiro de Banco de Dados. SBC, Porto Alegre, RS, Brasil, pp. 223–228, 2019.
Ren, X., Fu, X., Zhou, X., Liu, C., Gao, S., and Peng, L. Bilingual word embedding with sentence combination CNN for 1-to-n sentence alignment. In NLPIR 2020: 4th International Conference on Natural Language Processing and Information Retrieval, Seoul, Republic of Korea, December 18-20, 2020. ACM, pp. 119–124, 2020.
Ruder, S. An overview of gradient descent optimization algorithms. arXiv preprint arXiv:1609.04747 , 2016.
Schelter, S., Böse, J.-H., Kirschnick, J., Klein, T., and Seufert, S. Automatically tracking metadata and provenance of machine learning experiments. In ML Systems workshop, 2017.
Silva, V., Campos, V., Guedes, T., Camata, J. J., de Oliveira, D., Coutinho, A. L. G. A., Valduriez, P., and Mattoso, M. Dfanalyzer: Runtime dataflow analysis tool for computational science and engineering applications. SoftwareX vol. 12, pp. 100592, 2020.
Silva, V., de Oliveira, D., Mattoso, M., and Valduriez, P. Dfanalyzer: Runtime dataflow analysis of scientific applications using provenance. Proc. VLDB Endow. 11 (12): 2082–2085, 2018.
Silva, V., De Oliveira, D., Valduriez, P., and Mattoso, M. Analyzing related raw data files through dataflows. Concurrency and Computation: Practice and Experience 28 (8): 2528–2545, 2016.
Souza, R., Azevedo, L. G., Lourenço, V., Soares, E., Thiago, R., Brandão, R., Civitarese, D., Vital Brazil, E., Moreno, M., Valduriez, P., Mattoso, M., Cerqueira, R., and Netto, M. A. S. Workflow provenance in the lifecycle of scientific machine learning. Concurrency and Computation: Practice and Experience n/a (n/a): e6544, 2021.
Tsay, J., Mummert, T., Bobroff, N., Braz, A., Westerink, P., and Hirzel, M. Runway: machine learning model experiment management tool. In SysML, 2018.
Valero, M. Runtime aware architectures. In Proceedings of the 9th Annual Workshop on General Purpose Processing using Graphics Processing Unit, GPGPU@PPoPP 2016, Barcelona, Spain, March 12 - 16, 2016, D. R. Kaeli and J. Cavazos (Eds.). ACM, pp. 1, 2016.
Wang, D., Weisz, J. D., Muller, M., Ram, P., Geyer, W., Dugan, C., Tausczik, Y., Samulowitz, H., and Gray, A. Human-ai collaboration in data science: Exploring data scientists’ perceptions of automated ai. Proceedings of the ACM on Human-Computer Interaction 3 (CSCW): 1–24, 2019.
Yang, L., Meng, X., and Karniadakis, G. E. B-pinns: Bayesian physics-informed neural networks for forward and inverse PDE problems with noisy data. J. Comput. Phys. vol. 425, pp. 109913, 2021.
Zaharia, M., Chen, A., Davidson, A., Ghodsi, A., Hong, S. A., Konwinski, A., Murching, S., Nykodym, T., Ogilvie, P., Parkhe, M., Xie, F., and Zumar, C. Accelerating the machine learning lifecycle with mlflow. IEEE Data Eng. Bull. vol. 41, pp. 39–45, 2018.
Zeiler, M. D. Adadelta: an adaptive learning rate method. arXiv preprint arXiv:1212.5701 , 2012.
Zhu, Y. and Zabaras, N. Bayesian deep convolutional encoder–decoder networks for surrogate modeling and uncertainty quantification. Journal of Computational Physics vol. 366, pp. 415–447, 2018.
