Development of a real-time gesture recognition system for human-robot interaction

Ong, Niam Chi (2024) Development of a real-time gesture recognition system for human-robot interaction. Final Year Project, UTAR.

Preview

PDF Download (10Mb) | Preview

Abstract

In the quickly developing field of robotics and human-robot interaction (HRI), it is crucial for robots enable to recognize and react to human gestures in real-time. This study describes the creation and application of a real-time hand gesture detection system intended using TurtleBot3 Burger in Humble version to improve HRI's effectiveness and naturalness by leveraging recent developments in Robot Operating System (ROS2), computer vision, sensing, machine/deep learning, and Internet of Things (IoT). The system supports navigation and delivery tasks, monitors environmental temperature and humidity, captures images or records videos for surveillance and security, and integrates with Telegram for remote monitoring and alerts. To capture the finer details of hand gestures and ensure its supported functionality, the suggested system uses a multi-modal method that integrates data from laptop and raspberry pi cameras, and sensors such as LiDAR and DHT22. Robots can now understand a variety of gestures by detecting the number and sequence of open and closed fingers, thanks to the system's robust and accurate gesture detection, which is made possible by a carefully curated dataset and cutting-edge deep neural networks. Low latency between gesture input and robot reaction is made possible by effective model optimization and parallel processing, which gives the system its real-time characteristics. For fluid and interactive HRI situations including collaborative activities, assistive robotics, and entertainment applications, this real-time capacity is essential. The design architecture of the system, data pretreatment methods, and deep learning models used are discussed in the study, with an emphasis on the model's adaptation to various robot platforms and situations. Robots will be able to respond to human cues more contextually if natural language processing (NLP) techniques are incorporated to improve the contextual comprehension of gestures [2]. The system's great accuracy and robustness have been demonstrated through thorough testing in a variety of HRI settings. It has prospective applications in fields including home services, education, manufacturing, business, and entertainment where human-robot interaction must be natural and intuitive. In summary, the created real-time hand gesture detection system is a significant development in the field of HRI, allowing efficient and smooth communication between people and robots to bridge the gap between them. Its versatility and precision enable a wide range of real-world applications, potentially transforming how humans and robots collaborate and interact.

Actions (login required)