Abnormal behavior detection using intelligent video surveillance
Abstract
Intelligent surveillance systems have a large number of cameras installed. Abnormal vehicle or human entry at a certain location or time may potentially result in monetary loss and/or fatalities. This study develops a multi-surveillance camera intelligent surveillance system that is new, adaptable, and fast. The user may choose the number of interest zones with any polygon shape for each camera. Furthermore, the sort of abnormal item and the direction of abnormal motion for each location separately. To identify items in a video frame, the Single Shot Multi-Box Detector (SSD_MobileNet_v3) deep neural network is utilized. After that, these items are tracked using a Kernelized Correlation Filters (KCF) tracker in order to identify the direction of aberrant motion. Also, a novelty is to determine the people's motion type, i.e., running or walking, by establishing a relationship between the real human dimension and the observed distances in the video. The system's performance is evaluated on both the Authentically Distorted Surveillance Videos dataset and the newly collected dataset. An accuracy of 88.22% has been scored for event detection and F1-score of 87%. for people's motion classification. The experimental results confirm the superiority of the suggested method over the current state-of-the-art methods.
References
1. Kim, D., Kim, H., Mok, Y., & Paik, J. (2021, July 2). Real-Time Surveillance System for Analyzing Abnormal Behavior of Pedestrians. Applied Sciences, 11(13), 6153. https://doi.org/10.3390/app11136153.
2. George, M., Jose, B. R., & Mathew, J. (2018). Performance Evaluation of KCF based Trackers using VOT Dataset. Procedia Computer Science, 125, 560–567. https://doi.org/10.1016/j.procs.2017.12.072
3. Jian, Z., Yonghui, Z., Yan, Y., Ruonan, L., & Xueyao, W. (2020, November 28). MobileNet-SSD with adaptive expansion of receptive field. 2020 IEEE 3rd International Conference of Safe Production and Informatization(IICSPI). https://doi.org/10.1109/iicspi51290.2020.9332204
4. Zhao, Z., Lan, S., & Zhang, S. (2020, October). Human Pose Estimation based Speed Detection System for Running on Treadmill. 2020 International Conference on Culture-Oriented Science & Technology (ICCST). https://doi.org/10.1109/iccst50977.2020.00108.
5. Amrutha, C., Jyotsna, C., & Amudha, J. (2020, March). Deep Learning Approach for Suspicious Activity Detection from Surveillance Video. 2020 2nd International Conference on Innovative Mechanisms for Industry Applications (ICIMIA). https://doi.org/10.1109/icimia48430.2020.9074920.
6. Sun, C., Jia, Y., Song, H., & Wu, Y. (2021). Adversarial 3D Convolutional Auto-Encoder for Abnormal Event Detection in Videos. IEEE Transactions on Multimedia, 23,3292–3305. https://doi.org/10.1109/tmm.2020.3023303.
7. Bouindour, S., Hu, R., & Snoussi, H. (2019, June). Enhanced Convolutional Neural Network for Abnormal Event Detection in Video Streams. 2019 IEEE Second International Conference on Artificial Intelligence and Knowledge Engineering (AIKE). https://doi.org/10.1109/aike.2019.00039.
8. Kim, J. S., Kim, M. G., & Pan, S. B. (2021, November 21). A study on implementation of real-time intelligent video surveillance system based on embedded module. EURASIP Journal on Image and Video Processing, 2021(1). https://doi.org/10.1186/s13640-021-00576-0.
9. Luo, W., Liu, W., & Gao, S. (2017, July). Remembering history with convolutional LSTM for anomaly detection. 2017 IEEE International Conference on Multimedia and Expo(ICME). https://doi.org/10.1109/icme.2017.8019325.
10. Hasan, M., Choi, J., Neumann, J., Roy-Chowdhury, A. K., & Davis, L. S. (2016, June). Learning Temporal Regularity in Video Sequences. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). https://doi.org/10.1109/cvpr.2016.86.
11. Yu, J., Lee, Y., Yow, K. C., Jeon, M., & Pedrycz, W. (2022, August). Abnormal Event Detection and Localization via Adversarial Event Prediction. IEEE Transactions on Neural Networks and Learning Systems, 33(8),3572–3586. https://doi.org/10.1109/tnnls.2021.3053563.
12. Liu, C., Ying, J., Han, F., & Ruan, M. (2018, July). Abnormal Human Activity Recognition using Bayes Classifier and Convolutional Neural Network. 2018 IEEE 3rd International Conference on Signal and Image Processing(ICSIP). https://doi.org/10.1109/siprocess.2018.8600483
13. Kolekar, M. H., Bharti, N., & Patil, P. N. (2016, November). Detection of fence climbing using activity recognition by Support Vector Machine classifier. 2016 IEEE Region 10 Conference (TENCON). https://doi.org/10.1109/tencon.2016.7848029
14. Lao, S., Wang, D., li, F., & Zhang, H. (2016, December). Human running detection: Benchmark and baseline. Computer Vision and Image Understanding, 153, 143–150. https://doi.org/10.1016/j.cviu.2016.03.005
15. Seethi, V. D. R., & Bharti, P. (2020, September). CNN-based Speed Detection Algorithm for Walking and Running using Wrist-worn Wearable Sensors. 2020 IEEE International Conference on Smart Computing (SMARTCOMP). https://doi.org/10.1109/smartcomp50058.2020.00064
16. Ravanbakhsh, M., Nabi, M., Sangineto, E., Marcenaro, L., Regazzoni, C., & Sebe, N. (2017, September). Abnormal event detection in videos using generative adversarial nets. 2017 IEEE International Conference on Image Processing (ICIP). https://doi.org/10.1109/icip.2017.8296547.
17. Ganokratanaa, T., Aramvith, S., & Sebe, N. (2020). Unsupervised Anomaly Detection and Localization Based on Deep Spatiotemporal Translation Network. IEEE Access,8,50312–50329. https://doi.org/10.1109/access.2020.2979869.
18. Ghedia, N. S., & Vithalani, C. H. (2020, October 12). Outdoor object detection for surveillance based on modified GMM and Adaptive Thresholding. International Journal of Information Technology, 13(1), 185–193. https://doi.org/10.1007/s41870-020-00522-9
19. Verma, K. K., Singh, B. M., & Dixit, A. (2019, September 20). A review of supervised and unsupervised machine learning techniques for suspicious behavior recognition in intelligent surveillance system. International Journal of Information Technology, 14(1), 397–410. https://doi.org/10.1007/s41870-019-00364-0
20. Nandhini, C., & Brindha, M. (2022, August 23). Transfer learning based SSD model for helmet and multiple rider detection. International Journal of Information Technology,15(2),565–576. https://doi.org/10.1007/s41870-022-01058-w
21. Sreedhar, S., Philip, A. O., & Sreeja, M. U. (2023, August 24). Autotrack: a framework for query-based vehicle tracking and retrieval from CCTV footages using machine learning at the edge. International Journal of Information Technology, 15(7), 3827–3837. https://doi.org/10.1007/s41870-023-01415-3
22. Mishra, S., & Jabin, S. (2023, December 24). Anomaly detection in surveillance videos using deep autoencoder. International Journal of Information Technology,16(2),1111–1122. https://doi.org/10.1007/s41870-023-01659-z
23. Redmon, J. and Farhadi, A. (2018) YOLOv3: An Incremental Improvement. IEEE Conference on Computer Vision and Pattern Recognition, 89-95.
24. Liu, W., Anguelov, D., Erhan, D., Szegedy, C., Reed, S., Fu, C. Y., & Berg, A. C. (2016). SSD: Single Shot MultiBox Detector. Computer Vision – ECCV 2016, 21–37. https://doi.org/10.1007/978-3-319-46448-0_2
25. Ren, S., He, K., Girshick, R., & Sun, J. (2017, June 1). Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 39(6), 1137–1149. https://doi.org/10.1109/tpami.2016.2577031.
26. Andrew Howard, Mark Sandler, Grace Chu, Liang-Chieh Chen, Bo Chen, Mingxing Tan, (2019) Searching for MobileNetV3. arXiv:1905.02244v5 [cs.CV]
27. Panero, J., & Zelnik, M. (1979, November 1). Human Dimension and Interior Space. Watson-Guptill. http://books.google.ie/books?id=pQ1QAAAAMAAJ&q=)Human+Dimension+and+Interior+Space&dq=)Human+Dimension+and+Interior+Space&hl=&cd=2&source=gbs_api
28. Browning, R. C., Baker, E. A., Herron, J. A. and Kram, R. (2006). Effects of obesity and sex on the energetic cost and preferred speed of walking. Journal of Applied Physiology. 100 (2): 390–398.
29. Franco, C. A. (2022, May 18). Authentically Distorted Surveillance Videos Dataset. IEEE DataPort. https://ieee-dataport.org/open-access/authentically-distorted-surveillance-videos-dataset(Ren et al., 2017)
2. George, M., Jose, B. R., & Mathew, J. (2018). Performance Evaluation of KCF based Trackers using VOT Dataset. Procedia Computer Science, 125, 560–567. https://doi.org/10.1016/j.procs.2017.12.072
3. Jian, Z., Yonghui, Z., Yan, Y., Ruonan, L., & Xueyao, W. (2020, November 28). MobileNet-SSD with adaptive expansion of receptive field. 2020 IEEE 3rd International Conference of Safe Production and Informatization(IICSPI). https://doi.org/10.1109/iicspi51290.2020.9332204
4. Zhao, Z., Lan, S., & Zhang, S. (2020, October). Human Pose Estimation based Speed Detection System for Running on Treadmill. 2020 International Conference on Culture-Oriented Science & Technology (ICCST). https://doi.org/10.1109/iccst50977.2020.00108.
5. Amrutha, C., Jyotsna, C., & Amudha, J. (2020, March). Deep Learning Approach for Suspicious Activity Detection from Surveillance Video. 2020 2nd International Conference on Innovative Mechanisms for Industry Applications (ICIMIA). https://doi.org/10.1109/icimia48430.2020.9074920.
6. Sun, C., Jia, Y., Song, H., & Wu, Y. (2021). Adversarial 3D Convolutional Auto-Encoder for Abnormal Event Detection in Videos. IEEE Transactions on Multimedia, 23,3292–3305. https://doi.org/10.1109/tmm.2020.3023303.
7. Bouindour, S., Hu, R., & Snoussi, H. (2019, June). Enhanced Convolutional Neural Network for Abnormal Event Detection in Video Streams. 2019 IEEE Second International Conference on Artificial Intelligence and Knowledge Engineering (AIKE). https://doi.org/10.1109/aike.2019.00039.
8. Kim, J. S., Kim, M. G., & Pan, S. B. (2021, November 21). A study on implementation of real-time intelligent video surveillance system based on embedded module. EURASIP Journal on Image and Video Processing, 2021(1). https://doi.org/10.1186/s13640-021-00576-0.
9. Luo, W., Liu, W., & Gao, S. (2017, July). Remembering history with convolutional LSTM for anomaly detection. 2017 IEEE International Conference on Multimedia and Expo(ICME). https://doi.org/10.1109/icme.2017.8019325.
10. Hasan, M., Choi, J., Neumann, J., Roy-Chowdhury, A. K., & Davis, L. S. (2016, June). Learning Temporal Regularity in Video Sequences. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). https://doi.org/10.1109/cvpr.2016.86.
11. Yu, J., Lee, Y., Yow, K. C., Jeon, M., & Pedrycz, W. (2022, August). Abnormal Event Detection and Localization via Adversarial Event Prediction. IEEE Transactions on Neural Networks and Learning Systems, 33(8),3572–3586. https://doi.org/10.1109/tnnls.2021.3053563.
12. Liu, C., Ying, J., Han, F., & Ruan, M. (2018, July). Abnormal Human Activity Recognition using Bayes Classifier and Convolutional Neural Network. 2018 IEEE 3rd International Conference on Signal and Image Processing(ICSIP). https://doi.org/10.1109/siprocess.2018.8600483
13. Kolekar, M. H., Bharti, N., & Patil, P. N. (2016, November). Detection of fence climbing using activity recognition by Support Vector Machine classifier. 2016 IEEE Region 10 Conference (TENCON). https://doi.org/10.1109/tencon.2016.7848029
14. Lao, S., Wang, D., li, F., & Zhang, H. (2016, December). Human running detection: Benchmark and baseline. Computer Vision and Image Understanding, 153, 143–150. https://doi.org/10.1016/j.cviu.2016.03.005
15. Seethi, V. D. R., & Bharti, P. (2020, September). CNN-based Speed Detection Algorithm for Walking and Running using Wrist-worn Wearable Sensors. 2020 IEEE International Conference on Smart Computing (SMARTCOMP). https://doi.org/10.1109/smartcomp50058.2020.00064
16. Ravanbakhsh, M., Nabi, M., Sangineto, E., Marcenaro, L., Regazzoni, C., & Sebe, N. (2017, September). Abnormal event detection in videos using generative adversarial nets. 2017 IEEE International Conference on Image Processing (ICIP). https://doi.org/10.1109/icip.2017.8296547.
17. Ganokratanaa, T., Aramvith, S., & Sebe, N. (2020). Unsupervised Anomaly Detection and Localization Based on Deep Spatiotemporal Translation Network. IEEE Access,8,50312–50329. https://doi.org/10.1109/access.2020.2979869.
18. Ghedia, N. S., & Vithalani, C. H. (2020, October 12). Outdoor object detection for surveillance based on modified GMM and Adaptive Thresholding. International Journal of Information Technology, 13(1), 185–193. https://doi.org/10.1007/s41870-020-00522-9
19. Verma, K. K., Singh, B. M., & Dixit, A. (2019, September 20). A review of supervised and unsupervised machine learning techniques for suspicious behavior recognition in intelligent surveillance system. International Journal of Information Technology, 14(1), 397–410. https://doi.org/10.1007/s41870-019-00364-0
20. Nandhini, C., & Brindha, M. (2022, August 23). Transfer learning based SSD model for helmet and multiple rider detection. International Journal of Information Technology,15(2),565–576. https://doi.org/10.1007/s41870-022-01058-w
21. Sreedhar, S., Philip, A. O., & Sreeja, M. U. (2023, August 24). Autotrack: a framework for query-based vehicle tracking and retrieval from CCTV footages using machine learning at the edge. International Journal of Information Technology, 15(7), 3827–3837. https://doi.org/10.1007/s41870-023-01415-3
22. Mishra, S., & Jabin, S. (2023, December 24). Anomaly detection in surveillance videos using deep autoencoder. International Journal of Information Technology,16(2),1111–1122. https://doi.org/10.1007/s41870-023-01659-z
23. Redmon, J. and Farhadi, A. (2018) YOLOv3: An Incremental Improvement. IEEE Conference on Computer Vision and Pattern Recognition, 89-95.
24. Liu, W., Anguelov, D., Erhan, D., Szegedy, C., Reed, S., Fu, C. Y., & Berg, A. C. (2016). SSD: Single Shot MultiBox Detector. Computer Vision – ECCV 2016, 21–37. https://doi.org/10.1007/978-3-319-46448-0_2
25. Ren, S., He, K., Girshick, R., & Sun, J. (2017, June 1). Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 39(6), 1137–1149. https://doi.org/10.1109/tpami.2016.2577031.
26. Andrew Howard, Mark Sandler, Grace Chu, Liang-Chieh Chen, Bo Chen, Mingxing Tan, (2019) Searching for MobileNetV3. arXiv:1905.02244v5 [cs.CV]
27. Panero, J., & Zelnik, M. (1979, November 1). Human Dimension and Interior Space. Watson-Guptill. http://books.google.ie/books?id=pQ1QAAAAMAAJ&q=)Human+Dimension+and+Interior+Space&dq=)Human+Dimension+and+Interior+Space&hl=&cd=2&source=gbs_api
28. Browning, R. C., Baker, E. A., Herron, J. A. and Kram, R. (2006). Effects of obesity and sex on the energetic cost and preferred speed of walking. Journal of Applied Physiology. 100 (2): 390–398.
29. Franco, C. A. (2022, May 18). Authentically Distorted Surveillance Videos Dataset. IEEE DataPort. https://ieee-dataport.org/open-access/authentically-distorted-surveillance-videos-dataset(Ren et al., 2017)
Authors
Shaban, M., Elpeltagy , M. ., Y Khedr , A., & Al-Marakbey , A. (2024). Abnormal behavior detection using intelligent video surveillance. International Journal of Advanced Science and Computer Applications, 3(1). https://doi.org/10.47679/ijasca.v3i1.100
This work is licensed under a Creative Commons Attribution 4.0 International License.