Real-Time Computer Vision System Based on Convolutional Neural Networks for Precision Object Detection and Tracking in Collaborative Industrial Robot Applications

Anggit Wirasto; Khoirun Nisa; Titi Christiana

Authors

Anggit Wirasto Universitas Harapan Bangsa
Khoirun Nisa Universitas Harapan Bangsa
Titi Christiana Universitas Sains dan Teknologi kumputer

Keywords:

Collaborative robotics, Computer vision, Convolutional neural networks, Object detection, Real-time systems

Abstract

The increasing adoption of collaborative robots in modern manufacturing environments requires reliable perception systems that can ensure both safety and operational efficiency during human–robot collaboration. This study proposes a CNN-based real-time computer vision system for object and human detection in shared robotic workspaces. The research focuses on developing and evaluating a single-stage deep learning detection model optimized for real-time performance while maintaining high detection accuracy. The proposed methodology includes dataset preparation, model training using transfer learning, real-time system implementation, and comprehensive performance evaluation. Experimental results demonstrate that the developed system achieves high detection accuracy, as reflected by strong precision, recall, and mean Average Precision (mAP) values, while maintaining low inference latency suitable for real-time operation. The system consistently operates above real-time frame-rate thresholds, ensuring timely perception updates required for safety-related decision-making in collaborative robotic environments. Graphical and quantitative analyses further confirm the stability of inference performance under dynamic interaction scenarios involving human movement and multiple objects. Compared with existing approaches, the proposed system provides a balanced trade-off between accuracy and computational efficiency, making it practical for deployment in safety-aware human–robot collaboration scenarios. Overall, the findings indicate that CNN-based real-time object detection systems can effectively support perception and situational awareness in collaborative robotics, contributing to safer and more efficient industrial automation.

References

[1] E. Matheson, R. Minto, E. G. G. Zampieri, M. Faccio, and G. Rosati, “Human--robot collaboration in manufacturing applications: A review,” Robotics, vol. 8, no. 4, p. 100, 2019, doi: 10.3390/robotics8040100.

[2] A. S. M. Sahan, S. Kathiravan, M. Lokesh, and R. Raffik, “Role of cobots over industrial robots in Industry 5.0: A review,” in Proceedings of the 2nd International Conference on Advancements in Electrical, Electronics, Communication, Computing and Automation (ICAECA), 2023. doi: 10.1109/ICAECA56562.2023.10201199.

[3] A. D. S. Martin, L. F. R. Pinto, G. C. D. Oliveira Neto, and F. Facchini, “Collaborative robot in engine assembly: A socioeconomic approach to technological advancement in manufacturing,” IET Collab. Intell. Manuf., vol. 7, no. 1, p. e70044, 2025, doi: 10.1049/cim2.70044.

[4] Z. M. Bi, M. Luo, Z. Miao, B. Zhang, W. J. Zhang, and L. Wang, “Safety assurance mechanisms of collaborative robotic systems in manufacturing,” Robot. Comput. Integr. Manuf., vol. 67, p. 102022, 2021, doi: 10.1016/j.rcim.2020.102022.

[5] A. Latif, A. Mughall, M. H. D. Khan, and M. D. Khan, “A safety-enhancing framework based on collaborative robots (CoBot) for Industry 4.0,” in Proceedings of the 2024 International Conference on Engineering and Computing (ICECT), 2024. doi: 10.1109/ICECT61618.2024.10581298.

[6] K. P. Nguyen and Y. J. Ma, “Potential challenges of collaborative robot implementation in Vietnamese garment manufacturing,” IAES Int. J. Robot. Autom., vol. 13, no. 3, pp. 283–292, 2024, doi: 10.11591/ijra.v13i3.pp283-292.

[7] R. Shah, A. S. A. Doss, and N. Lakshmaiya, “Advancements in AI-enhanced collaborative robotics: Towards safer, smarter, and human-centric industrial automation,” Results Eng., vol. 27, p. 105704, 2025, doi: 10.1016/j.rineng.2025.105704.

[8] P. Goyal et al., “Mechanisms for ensuring the security of collaborative robot systems in industrial settings,” Multidiscip. Rev., vol. 8, p. e2025ss0106, 2025, doi: 10.31893/multirev.2025ss0106.

[9] J. D. A. Dornelles, N. F. Ayala, and A. G. Frank, “Collaborative or substitutive robots? Effects on workers’ skills in manufacturing activities,” Int. J. Prod. Res., vol. 61, no. 22, pp. 7922–7955, 2023, doi: 10.1080/00207543.2023.2240912.

[10] D. Chu, S. Yu, Y. Ling, Y. Zhao, and J. Zhang, “A game-theoretic and multimodal interaction framework for collaborative robots in smart manufacturing,” Decis. Mak. Appl. Manag. Eng., vol. 8, no. 2, pp. 36–52, 2025, doi: 10.31181/dmame8220251475.

[11] I. Rybalskii, K. Kruusamäe, A. K. Singh, and S. Schlund, “An augmented reality interface for safer human--robot interaction in manufacturing,” in IFAC-PapersOnLine, Elsevier, 2024, pp. 581–585. doi: 10.1016/j.ifacol.2024.09.275.

[12] B. Darmanin, A. Bonello, and E. Francalanza, “A systematic design approach for cognitively ergonomic collaborative robotic workspaces,” Procedia CIRP, vol. 130, pp. 853–860, 2024, doi: 10.1016/j.procir.2024.10.175.

[13] R.-J. Halme, M. Lanz, J. Kämäräinen, R. Pieters, J. Latokartano, and A. Hietanen, “Review of vision-based safety systems for human-robot collaboration,” Procedia CIRP, vol. 72, pp. 111–116, 2018, doi: 10.1016/j.procir.2018.03.043.

[14] Y. Cohen, A. Biton, and S. Shoval, “Fusion of computer vision and AI in collaborative robotics: A review and future prospects,” Appl. Sci., vol. 15, no. 14, p. 7905, 2025, doi: 10.3390/app15147905.

[15] S. R. Addula and A. K. Tyagi, “Future of computer vision and industrial robotics in smart manufacturing,” in Artificial intelligence-enabled digital twin for smart manufacturing, Wiley, 2025, pp. 505–539. doi: 10.1002/9781394303601.ch22.

[16] L. M. Amaya-Mejia, N. Duque-Suarez, D. Jaramillo-Ramirez, and C. Martinez, “Vision-based safety system for barrierless human-robot collaboration,” in 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), IEEE, 2022, pp. 7331–7336. doi: 10.1109/IROS47612.2022.9981689.

[17] M. Forlini, F. Neri, C. Scoccia, L. Carbonari, and G. Palmieri, “Collision avoidance in collaborative robotics based on real-time skeleton tracking,” in Mechanisms and Machine Science, Springer, 2023, pp. 81–88. doi: 10.1007/978-3-031-32606-6_10.

[18] J. Humphries, P. de Ven, N. Amer, N. Nandeshwar, and A. Ryan, “Managing safety of the human on the factory floor: A computer vision fusion approach,” Technol. Sustain., vol. 3, no. 3, pp. 309–331, 2024, doi: 10.1108/TECHS-12-2023-0054.

[19] M. J. Alenjareghi, S. Keivanpour, Y. A. Chinniah, and S. Jocelyn, “Computer vision-enabled real-time job hazard analysis for safe human--robot collaboration in disassembly tasks,” J. Intell. Manuf., vol. 36, no. 8, pp. 5563–5591, 2025, doi: 10.1007/s10845-024-02519-8.

[20] B. Malobický et al., “Towards seamless human--robot interaction: Integrating computer vision for tool handover and gesture-based control,” Appl. Sci., vol. 15, no. 7, p. 3575, 2025, doi: 10.3390/app15073575.

[21] Y. Liu and H. Jebelli, “Intention estimation in physical human-robot interaction in construction: Empowering robots to gauge workers’ posture,” in Construction Research Congress 2022: Computer Applications, Automation, and Data Analytics, ASCE, 2022, pp. 621–630. doi: 10.1061/9780784483961.065.

[22] J. L. Crowley, “Convolutional neural networks,” in Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 13500 LNAI, Springer, 2023, pp. 67–80. doi: 10.1007/978-3-031-24349-3_5.

[23] P. Shruti and R. Rekha, “A review of convolutional neural networks, its variants and applications,” in Proceedings of the 2023 International Conference on Intelligent Systems for Communication, IoT and Security (ICISCoIS 2023), IEEE, 2023, pp. 31–36. doi: 10.1109/ICISCoIS56541.2023.10100412.

[24] P. Kumari, “Transforming medical imaging with convolutional neural networks (CNNs): Advances in diagnosis and treatment,” in Deep learning in medical signal and image processing, 2025, pp. 195–230. doi: 10.4018/979-8-3693-9816-6.ch009.

[25] E. Arkin, N. Yadikar, Y. Muhtar, and K. Ubul, “A survey of object detection based on CNN and transformer,” in 2021 IEEE 2nd International Conference on Pattern Recognition and Machine Learning (PRML 2021), IEEE, 2021, pp. 99–108. doi: 10.1109/PRML52754.2021.9520732.

[26] M. Sornalakshmi, M. Sakthimohan, G. Elizabeth Rani, V. Aravindhan, B. K. Surya, and M. Devadharshni, “Real time object detection using deep learning,” in ViTECoN 2023---2nd IEEE International Conference on Vision Towards Emerging Trends in Communication and Networking Technologies, Proceedings, IEEE, 2023. doi: 10.1109/ViTECoN58111.2023.10157311.

[27] T. M. Geethanjali et al., “Real time object detection & recognition: A comparative study of YOLOv3 and YOLOv7 in OpenCV,” in 15th International Conference on Advances in Computing, Control, and Telecommunication Technologies (ACT 2024), 2024, pp. 6627–6637.

[28] A. Tanisha, N. Tanisha, M. Chaitra, and P. R. Tejasri, “Real-time object detection from surveillance using deep learning,” in Proceedings of the 3rd International Conference on Intelligent and Innovative Technologies in Computing, Electrical and Electronics (IITCEE 2025), IEEE, 2025. doi: 10.1109/IITCEE64140.2025.10915303.

[29] M. Qi, Z. Bin, H. Wang, B. Xie, F. Xiang, and Z. Chen, “Evaluation of real-time object detection model based on small targets,” in Proceedings of SPIE---The International Society for Optical Engineering, 2019, p. 108430M. doi: 10.1117/12.2505930.

[30] N. C. Francis and J. M. Mathana, “Survey on object detection in VLSI architecture through deep learning,” in AIP Conference Proceedings, AIP Publishing, 2024, p. 30019. doi: 10.1063/5.0209400.

[31] S. Ram and A. Gupta, “Pre-trained deep networks for faster region-based CNN model for pituitary tumor detection,” in Lecture Notes in Networks and Systems, Springer, 2021, pp. 479–498. doi: 10.1007/978-981-33-4355-9_36.

[32] D. Danang, A. B. Santoso, and M. U. Dewi, “CICA Framework: Harnessing CSR, AI, and Blockchain for Sustainable Digital Culture,” Int. J. Adv. Comput. Sci. & Appl., vol. 16, no. 11, 2025.

[33] D. Danang, T. Wahyono, I. Sembiring, T. Wellem, and N. H. Dzulkefly, “An Adaptive Framework Integrating ML Blockchain and TEE for Cloud Security,” in 2025 4th International Conference on Creative Communication and Innovative Technology (ICCIT), 2025, pp. 1–7.

[34] D. Danang, I. A. Dianta, A. B. Santoso, and S. Kholifah, “Hybrid CNN GRU Framework for Early Detection and Adaptive Mitigation of DDoS Attacks in SDN using Image Based Traffic Analysis,” Int. J. Inf. Eng. Sci., vol. 2, no. 2, pp. 66–78, 2025, doi: 10.62951/ijies.v2i2.292.

[35] D. Danang, M. U. Dewi, and G. Widhiati, “Federated Hybrid CNN GRU and COBCO Optimized Elman Neural Network for Real Time DDoS Detection in Cloud Edge Environments,” Int. J. Electr. Eng. Math. Comput. Sci., vol. 2, no. 2, pp. 28–35, 2025, doi: 10.62951/ijeemcs.v2i2.293.

[36] D. Danang, S. Siswanto, W. Aryani, and P. Wibowo, “Hybrid Federated Ensemble Learning Approach for Real-Time Distributed DDoS Detection in IIoT Edge Computing Environment,” J. Eng. Electr. Informatics, vol. 5, no. 1, pp. 9–17, 2025, doi: 10.55606/jeei.v5i1.5099.

[37] D. Danang, N. D. Setiawan, and E. Siswanto, “Pemanfaatan Teknologi Internet of Things untuk Monitoring Kualitas Air Sungai di Wilayah Perkotaan,” J. New Trends Sci., vol. 2, no. 1, pp. 23–34, 2024.

[38] E. Muhadi, S. Sulartopo, D. Danang, D. Sasmoko, and N. D. Setiawan, “Rancang bangun sistem keamanan ruang persandian menggunakan RFID dan sensor PIR berbasis IOT,” Router J. Tek. Inform. dan Terap., vol. 2, no. 1, pp. 8–20, 2024.

[39] M. K. Umam, D. Danang, E. Siswanto, and N. D. Setiawan, “Rancangan Bangun Otomasi Air Suling Daun Cengkeh Berbasis Arduino,” Repeater Publ. Tek. Inform. dan Jar., vol. 2, no. 2, pp. 1–10, 2024.

[40] H. R. Putranti, R. Retnowati, A. A. Sihombing, and D. Danang, “Performance assessment through work gamification: Investigating engagement,” South African J. Bus. Manag., vol. 55, no. 1, pp. 1–12, 2024.

[41] I. Englishtina, H. R. D. Putranti, D. Danang, and A. A. B. Pujiati, “SITENAR CERYA as an innovation in English language learning at SMP Stella Matutina Salatiga: Merging technology and folktales,” REKA ELKOMIKA J. Pengabdi. Kpd. Masy., vol. 5, no. 3, pp. 241–250, 2024.

[42] H. R. D. Putranti, D. Danang, T. Da Silva, and A. A. B. Pujiati, “Integrating Hands-on and Virtual Learning for Environmental Sustainability: Eco Enzyme Soap Making at Stella Matutina,” REKA ELKOMIKA J. Pengabdi. Kpd. Masy., vol. 6, no. 1, pp. 88–97, 2025.