Evaluating Temporal Bias in Time Series Event Detection Methods
DOI:
https://doi.org/10.5753/jidm.2021.1968Keywords:
Event Detection, Time Series, Benchmarking, Temporal BiasAbstract
The detection of events in time series is an important task in several areas of knowledge where operations monitoring is essential. Experts often have to deal with choosing the most appropriate event detection method for a time series, which can be a complex task. There is a demand for benchmarking different methods in order to guide this choice. For this, standard classification accuracy metrics are usually adopted. However, they are insufficient for a qualitative analysis of the tendency of a method to precede or delay event detections. Such analysis is interesting for applications in which tolerance for "close" detections is important rather than focusing only on accurate ones. In this context, this paper proposes a more comprehensive event detection benchmark process, including an analysis of temporal bias of detection methods. For that, metrics based on the time distance between event detections and identified events (detection delay) are adopted. Computational experiments were conducted using real-world and synthetic datasets from Yahoo Labs and resources from the Harbinger framework for event detection. Adopting the proposed detection delay-based metrics helped obtain a complete overview of the performance and general behavior of detection methods.
Downloads
References
Agrawal, S. and Agrawal, J. Survey on anomaly detection using data mining techniques. In Procedia Computer Science. Vol. 60. pp. 708–713, 2015.
Aminikhanghahi, S. and Cook, D. A survey of methods for time series change point detection. Knowledge and Information Systems 51 (2): 339–367, 2017.
Assareh, H., Smith, I., and Mengersen, K. Change point detection in risk adjusted control charts. Statistical Methods in Medical Research 24 (6): 747–768, 2015.
Atashgar, K., Rafiee, N., and Karbasian, M. A new hybrid approach to panel data change point detection. Communications in Statistics - Theory and Methods, 2020.
Braei, M. and Wagner, S. Anomaly Detection in Univariate Time-series: A Survey on the State-of-the-Art. arXiv:2004.00433 [cs, stat] , Apr., 2020.
Carmona, R. Statistical Analysis of Financial Data in R. Springer Science & Business Media, 2013.
Chandola, V., Banerjee, A., and Kumar, V. Anomaly detection: A survey. ACM Computing Surveys 41 (3), 2009.
Chauhan, V., Dahiya, K., and Sharma, A. Problem formulations and solvers in linear SVM: a review. Artificial Intelligence Review 52 (2): 803–855, 2019.
Chen, H. and Zhang, N. Graph-based change-point detection. Annals of Statistics 43 (1): 139–176, 2015.
Dong, C., Jin, B., and Li, D. Predicting the heating value of MSW with a feed forward neural network. Waste Management 23 (2): 103–106, 2003.
Gammerman, A. and Vovk, V. Hedging predictions in machine learning. The Computer Journal 50 (2): 151–163, 2007.
Greff, K., Srivastava, R. K., Koutník, J., Steunebrink, B. R., and Schmidhuber, J. Lstm: A search space odyssey. IEEE Transactions on Neural Networks and Learning Systems 28 (10): 2222–2232, 2017.
Guo, T., Dong, J., Li, H., and Gao, Y. Simple convolutional neural network on image classification. In 2017 IEEE 2nd International Conference on Big Data Analysis, ICBDA 2017. pp. 721–724, 2017.
Gupta, M., Gao, J., Aggarwal, C., and Han, J. Outlier Detection for Temporal Data: A Survey. IEEE Transactions on Knowledge and Data Engineering 26 (9): 2250–2267, 2014.
Gupta, V., Gupta, A., Kumar, D., and Sardana, A. Prediction of COVID-19 confirmed, death, and cured cases in India using random forest model. Big Data Mining and Analytics 4 (2): 116–123, 2021.
Guralnik, V. and Srivastava, J. Event Detection from Time Series Data. In Proceedings of the Fifth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’99. ACM, New York, NY, USA, pp. 33–42, 1999.
Han, J., Pei, J., and Kamber, M. Data Mining: Concepts and Techniques. Elsevier, 2011.
Haykin, S. O. Neural Networks and Learning Machines. Pearson Education, 2011.
Huang, G.-B., Zhou, H., Ding, X., and Zhang, R. Extreme learning machine for regression and multiclass classification. IEEE Transactions on Systems, Man, and Cybernetics, Part B: Cybernetics 42 (2): 513–529, 2012.
Huang, G.-B., Zhu, Q.-Y., and Siew, C.-K. Extreme learning machine: Theory and applications. Neurocomputing 70 (1-3): 489–501, 2006.
Ismaeel, S., Miri, A., and Chourishi, D. Using the Extreme Learning Machine (ELM) technique for heart disease diagnosis. In 2015 IEEE Canada International Humanitarian Technology Conference, IHTC 2015, 2015.
Lavin, A. and Ahmad, S. Evaluating real-time anomaly detection algorithms - The numenta anomaly benchmark. In Proceedings - 2015 IEEE 14th International Conference on Machine Learning and Applications, ICMLA 2015. pp. 38–44, 2016.
Lim, B. and Zohren, S. Time-series forecasting with deep learning: A survey. Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences 379 (2194), 2021.
Muniyandi, A., Rajeswari, R., and Rajaram, R. Network anomaly detection by cascading k-Means clustering and C4.5 decision tree algorithm. In Procedia Engineering. Vol. 30. pp. 174–182,2012.
Rahul and Choudhary, B. An Advanced Genetic Algorithm with Improved Support Vector Machine for Multi-Class Classification of Real Power Quality Events. Electric Power Systems Research vol. 191, 2021.
Raza, H., Prasad, G., and Li, Y. EWMA model based shift-detection methods for detecting covariate shifts in non-stationary environments. Pattern Recognition 48 (3): 659–669, 2015.
Riese, F. and Keller, S. Supervised, semi-supervised, and unsupervised learning for hyperspectral regression. Advances in Computer Vision and Pattern Recognition, 2020.
Salles, R., Belloze, K., Porto, F., Gonzalez, P., and Ogasawara, E. Nonstationary time series transformation methods: An experimental review. Knowledge-Based Systems vol. 164, pp. 274–291, 2019.
Salles, R., Escobar, L., Baroni, L., Zorrilla, R., Ziviani, A., Kreischer, V., Delicato, F., Pires, P. F., Maia, L., Coutinho, R., Assis, L., and Ogasawara, E. Harbinger: Um framework para integração e análise de métodos de detecção de eventos em séries temporais. In Anais do Simpósio Brasileiro de Banco de Dados (SBBD). SBC, pp. 73–84, 2020.
Singh, N. and Olinsky, C. Demystifying Numenta anomaly benchmark. In Proceedings of the International Joint Conference on Neural Networks. Vol. 2017-May. pp. 1570–1577, 2017.
Takeuchi, J.-I. and Yamanishi, K. A unifying framework for detecting outliers and change points from time series. IEEE Transactions on Knowledge and Data Engineering 18 (4): 482–492, 2006.
Tang, J., Deng, C., and Huang, G.-B. Extreme Learning Machine for Multilayer Perceptron. IEEE Transactions on Neural Networks and Learning Systems 27 (4): 809–821, 2016.
Wang, Y., Vuran, M., and Goddard, S. Analysis of event detection delay in wireless sensor networks. In Proceedings - IEEE INFOCOM. pp. 1296–1304, 2011.
Webscope, Y. Labeled anomaly detection dataset, 2015.
Zhang, J., Jiang, R., Li, B., and Xu, N. An automatic recognition method of microseismic signals based on EEMD-SVD and ELM. Computers and Geosciences vol. 133, 2019.
