A Hybrid NeuralSymbolic Approach for Human Robot Interaction Enhancement Using Multimodal Sensor Fusion and Context Aware Behavioral Adaptation Techniques

Setyawan Wibisono; Hayadi Hamuda; Encik Yoega Renaldi

Authors

Setyawan Wibisono Universitas Stikubank
Hayadi Hamuda Universitas Pamulang
Encik Yoega Renaldi Universitas Pamulang

Keywords:

Context-Aware Reasoning, Human Robot Interaction, Hybrid Neural–Symbolic Systems, Interpretability, Multimodal Perception

Abstract

Human–Robot Interaction (HRI) systems increasingly rely on data-driven approaches to interpret multimodal sensory inputs and support natural interaction. However, purely neural-based HRI models often suffer from limited interpretability and insufficient context-aware decision-making, which can reduce user trust and adaptability in dynamic interaction scenarios. To address these limitations, this study proposes a hybrid neural–symbolic HRI framework that integrates multimodal neural perception with explicit symbolic reasoning for adaptive and interpretable robot behavior. The proposed system combines deep neural networks for processing visual, speech, and gesture inputs with a rule-based symbolic reasoning layer that models interaction context, user states, and behavioral constraints. A loosely coupled integration strategy enables neural outputs to be transformed into symbolic representations, allowing logical inference to guide action selection while preserving perceptual accuracy. The framework was evaluated through controlled HRI experiments comparing a neural-only baseline with the proposed hybrid configuration across multiple interaction scenarios. Experimental results demonstrate that the hybrid neural–symbolic system significantly improves interaction accuracy, contextual responsiveness, and user satisfaction, while achieving substantial gains in interpretability. These findings indicate that symbolic reasoning effectively complements neural perception by enhancing transparency and context-aware adaptation without compromising performance. The study concludes that hybrid neural–symbolic architectures provide a promising foundation for developing trustworthy, adaptive, and human-centered HRI systems.

References

[1] T. Pekarek Rosin et al., “A Framework for Adapting Human-Robot Interaction to Diverse User Groups,” in Lecture Notes in Computer Science, 2025, pp. 24–38. doi: 10.1007/978-981-96-3525-2_3.

[2] I. Kishor, U. Mamodiya, S. Saini, and B. Bossoufi, “Voice-Enabled Human-Robot Interaction: Adaptive Self-Learning Systems for Enhanced Collaboration,” Robotica, vol. 43, no. 6, pp. 2143–2171, 2025, doi: 10.1017/S0263574725000438.

[3] O. Nocentini, L. Fiorini, G. Acerbi, A. Sorrentino, G. Mancioppi, and F. Cavallo, “A Survey of Behavioral Models for Social Robots,” Robotics, vol. 8, no. 3, p. 54, 2019, doi: 10.3390/robotics8030054.

[4] J. D. Kumar, N. Bansal, A. Kaushik, and A. Sethi, “Human-Robot Interaction: Designing Effective Interfaces for Collaborative Tasks,” in Proceedings of the IEEE International Conference on Advances in Computing, Communication and Networking, 2024, pp. 356–360. doi: 10.1109/ICAC2N63387.2024.10895441.

[5] J. Zhang and M. Lau, “Human-Robot Interaction: A {ROS2}-Based Approach with Voice and Vision Integration,” in Proceedings of the International Conference on Automation, Robotics and Computer Engineering, 2024, pp. 21–25. doi: 10.1109/ICARCE63054.2024.00011.

[6] D. K. Jain, A. Rahate, G. Joshi, R. Walambe, and K. Kotecha, “Employing Co-Learning to Evaluate the Explainability of Multimodal Sentiment Analysis,” IEEE Trans. Comput. Soc. Syst., vol. 11, no. 4, pp. 4673–4680, 2024, doi: 10.1109/TCSS.2022.3176403.

[7] M. Munir, S. A. Siddiqui, F. Küsters, D. Mercier, A. Dengel, and S. Ahmed, “{TSXplain:} Demystification of {DNN} Decisions for Time-Series Using Natural Language and Statistical Features,” in Lecture Notes in Computer Science, 2019, pp. 426–439. doi: 10.1007/978-3-030-30493-5_43.

[8] C. He, M. Ma, and P. Wang, “Extract Interpretability-Accuracy Balanced Rules from Artificial Neural Networks: A Review,” Neurocomputing, vol. 387, pp. 346–358, 2020, doi: 10.1016/j.neucom.2020.01.036.

[9] H. Wang, Y.-T. Huang, Y.-T. Xia, Z.-Z. Fan, G.-L. Luo, and H. Yang, “Multimodal Interpretable Image Classification Method Based on Visual Attributes,” Acta Autom. Sin., vol. 51, no. 2, pp. 445–456, 2025, doi: 10.16383/j.aas.c240218.

[10] Z. M. Arkah, B. Pontes, and C. Rubio, “Interpretation of Diabetic Foot Ulcer Image Classification Using Layer Attribution Algorithms,” in Lecture Notes in Networks and Systems, 2024, pp. 13–22. doi: 10.1007/978-3-031-75013-7_2.

[11] R. Vinjamuri, Discovering the Frontiers of Human-Robot Interaction: Insights and Innovations in Collaboration, Communication, and Control. Springer, 2024. doi: 10.1007/978-3-031-66656-8.

[12] B. N. De Carolis, C. Gena, A. Lieto, S. Rossi, and A. Sciutti, “3rd Workshop on Adapted Interaction with Social Robots,” in Proceedings of the 30th ACM Conference on User Modeling, Adaptation and Personalization, 2022. doi: 10.1145/3511047.3536345.

[13] X. Wang, H. Shen, H. Yu, J. Guo, and X. Wei, “Hand and Arm Gesture-Based Human-Robot Interaction: A Review,” in ACM International Conference Proceeding Series, 2022. doi: 10.1145/3564982.3564996.

[14] H. Wang, X. Li, and X. Zhang, “Multimodal Human-Robot Interaction on Service Robot,” in Proceedings of the IEEE Advanced Information Technology, Electronic and Automation Control Conference, 2021, pp. 2290–2295. doi: 10.1109/IAEAC50856.2021.9391068.

[15] D. Saha, N. K. Jisy, R. Vinu, S. S. Sivasankari, and A. Balodi, “Dual-Mode Robotic Arm Control System: Voice and Gesture Integration,” in Proceedings of the IEEE International Conference on Networks, Multimedia and Information Technology, 2025. doi: 10.1109/NMITCON65824.2025.11188250.

[16] U. Widelska, J. M. Moczydłowska, and A. Tul-Krzyszczuk, “Human-Robot Interaction: Ethical and Communicative Determinants of Trust,” in Trust in Generative Artificial Intelligence: Human-Robot Interaction and Ethical Considerations, Routledge, 2025, pp. 63–72. doi: 10.4324/9781003586937-8.

[17] E. Lee, E. Kim, and A. J. Choi, “Robotics in Metaverse,” in Human-Centered Metaverse: Concepts, Methods, and Applications, Elsevier, 2024, pp. 153–168. doi: 10.1016/B978-0-443-21996-2.00003-6.

[18] H. Tan, Y. Zhou, Q. Tao, J. Rosen, and S. van Dijken, “Bioinspired Multisensory Neural Network with Crossmodal Integration and Recognition,” Nat. Commun., vol. 12, no. 1, p. 1120, 2021, doi: 10.1038/s41467-021-21404-z.

[19] H. Yu and Q. Zhao, “Brain-Inspired Multisensory Integration Neural Network for Cross-Modal Recognition Through Spatio-Temporal Dynamics and Deep Learning,” Cogn. Neurodyn., vol. 18, no. 6, pp. 3615–3628, 2024, doi: 10.1007/s11571-023-09932-4.

[20] X. Zhou, S. Lan, W. Wang, X. Li, S. Zhou, and H. Yang, “Visual-Haptic-Kinesthetic Object Recognition with Multimodal Transformer,” in Lecture Notes in Computer Science, 2023, pp. 233–245. doi: 10.1007/978-3-031-44195-0_20.

[21] R. Calegari, G. Ciatto, S. Mariani, E. Denti, and A. Omicini, “{LPaaS} as Micro-Intelligence: Enhancing {IoT} with Symbolic Reasoning,” Big Data Cogn. Comput., vol. 2, no. 3, p. 23, 2018, doi: 10.3390/bdcc2030023.

[22] P. Hitzler, F. Bianchi, M. Ebrahimi, and M. K. Sarker, “Neural-Symbolic Integration and the Semantic Web,” Semant. Web, vol. 11, no. 1, pp. 3–11, 2020, doi: 10.3233/SW-190368.

[23] N. E. Zamri, S. S. M. Sidik, M. S. Mohd Kasihmuddin, M. A. Mansor, and N. A. Romli, “Weighted Random 2 Satisfiability Logic Training in Discrete Hopfield Neural Network via Election Algorithm,” in Intelligent Systems of Computing and Informatics in Sustainable Urban Development, 2025, pp. 43–58. doi: 10.1201/9781003518396-4.

[24] K. Yang and J. Deng, “Learning Symbolic Rules for Reasoning in Quasi-Natural Language,” Trans. Mach. Learn. Res., 2023.

[25] M. Sridevi et al., “Neurosymbolic {AI} for Context-Aware Resource Management in {5G} Smart Healthcare Networks,” J. Environ. Prot. Ecol., vol. 26, no. 3, pp. 1061–1071, 2025.

[26] P. Chandre, P. Mahalle, G. Shinde, B. Shendkar, and S. Kashid, “Neuro-Symbolic {AI}: A Future of Tomorrow,” {ASEAN} J. Sci. Technol. Dev., vol. 42, no. 2, pp. 187–200, 2025, doi: 10.61931/2224-9028.1620.

[27] M. H. Wang, “Explainable Artificial Intelligence Framework for Predicting Treatment Outcomes in Age-Related Macular Degeneration,” Sensors, vol. 25, no. 22, p. 6879, 2025, doi: 10.3390/s25226879.

[28] U. Nawaz, M. Anees-ur-Rahaman, and Z. Saeed, “A Review of Neuro-Symbolic {AI} Integrating Reasoning and Learning for Advanced Cognitive Systems,” Intell. Syst. with Appl., vol. 26, p. 200541, 2025, doi: 10.1016/j.iswa.2025.200541.

[29] H. Lemos, P. Avelar, M. Prates, A. Garcez, and L. Lamb, “Neural-Symbolic Relational Reasoning on Graph Models: Effective Link Inference and Computation from Knowledge Bases,” in Lecture Notes in Computer Science, 2020, pp. 647–659. doi: 10.1007/978-3-030-61609-0_51.

[30] A. Potapov, A. Belikov, V. Bogdanov, and A. Scherbatiy, “Cognitive Module Networks for Grounded Reasoning,” in Lecture Notes in Computer Science, 2019, pp. 148–158. doi: 10.1007/978-3-030-27005-6_15.

[31] A. Daniele, T. Campari, S. Malhotra, and L. Serafini, “Simple and Effective Transfer Learning for Neuro-Symbolic Integration,” in Lecture Notes in Computer Science, 2024, pp. 166–179. doi: 10.1007/978-3-031-71167-1_9.

[32] N. Manginas, G. Paliouras, and L. De Raedt, “{NeSyA:} Neurosymbolic Automata,” in Proceedings of the International Joint Conference on Artificial Intelligence (IJCAI 2025), 2025, pp. 5950–5958. doi: 10.24963/ijcai.2025/662.

[33] Y. Aspis, K. Broda, J. Lobo, and A. Russo, “{Embed2Sym}---Scalable Neuro-Symbolic Reasoning via Clustered Embeddings,” in Proceedings of the 19th International Conference on Principles of Knowledge Representation and Reasoning (KR 2022), 2022, pp. 421–431.

[34] S. Bartl, K. Innerebner, and E. Lex, “Differentiable Fuzzy Neural Networks for Recommender Systems,” in Adjunct Proceedings of the 33rd ACM Conference on User Modeling, Adaptation and Personalization (UMAP 2025), 2025, pp. 343–348. doi: 10.1145/3708319.3734174.

[35] D. Danang, S. Siswanto, W. Aryani, and P. Wibowo, “Hybrid Federated Ensemble Learning Approach for Real-Time Distributed DDoS Detection in IIoT Edge Computing Environment,” J. Eng. Electr. Informatics, vol. 5, no. 1, pp. 9–17, 2025, doi: 10.55606/jeei.v5i1.5099.

[36] E. Muhadi, S. Sulartopo, D. Danang, D. Sasmoko, and N. D. Setiawan, “Rancang bangun sistem keamanan ruang persandian menggunakan RFID dan sensor PIR berbasis IOT,” Router J. Tek. Inform. dan Terap., vol. 2, no. 1, pp. 8–20, 2024.

[37] D. Danang, I. A. Dianta, A. B. Santoso, and S. Kholifah, “Hybrid CNN GRU Framework for Early Detection and Adaptive Mitigation of DDoS Attacks in SDN using Image Based Traffic Analysis,” Int. J. Inf. Eng. Sci., vol. 2, no. 2, pp. 66–78, 2025, doi: 10.62951/ijies.v2i2.292.

[38] D. Danang, M. U. Dewi, and G. Widhiati, “Federated Hybrid CNN GRU and COBCO Optimized Elman Neural Network for Real Time DDoS Detection in Cloud Edge Environments,” Int. J. Electr. Eng. Math. Comput. Sci., vol. 2, no. 2, pp. 28–35, 2025, doi: 10.62951/ijeemcs.v2i2.293.

[39] I. Englishtina, H. R. D. Putranti, D. Danang, and A. A. B. Pujiati, “SITENAR CERYA as an innovation in English language learning at SMP Stella Matutina Salatiga: Merging technology and folktales,” REKA ELKOMIKA J. Pengabdi. Kpd. Masy., vol. 5, no. 3, pp. 241–250, 2024.

[40] H. R. D. Putranti, D. Danang, T. Da Silva, and A. A. B. Pujiati, “Integrating Hands-on and Virtual Learning for Environmental Sustainability: Eco Enzyme Soap Making at Stella Matutina,” REKA ELKOMIKA J. Pengabdi. Kpd. Masy., vol. 6, no. 1, pp. 88–97, 2025.

[41] D. Danang, A. B. Santoso, and M. U. Dewi, “CICA Framework: Harnessing CSR, AI, and Blockchain for Sustainable Digital Culture,” Int. J. Adv. Comput. Sci. & Appl., vol. 16, no. 11, 2025.

[42] D. Danang, H. Haryani, Q. Aini, F. A. Ramahdan, and J. Edwards, “Empowering digital literacy through blockchain based alphasign for secure and sustainable e-governance,” 2025.