Development of a hand gesture recognition using mediapipe landmarks for windows control system

Wong, Jenn Sen (2025) Development of a hand gesture recognition using mediapipe landmarks for windows control system. Final Year Project, UTAR.

PDF Download (3447Kb)

Abstract

This project presents the development of a real-time hand gesture recognition system aimed at enable for a intuitive and touchless human-computer interaction. This study explores 2 approaches: a hybrid method combining Mediapipe for hand landmark extraction with Convolutional Neural Network (CNN) classification, and an image-based approach using YOLOv8 for gesture classification. A dataset of nine predefined gestures was prepared with augmentation techniques such as flipping, rotation, scaling and noise addition to improve model generalization. The CNN-Mediapipe approach achieved a high validation accuracy and demonstrated superior performance in real-time execution, along with using PyAutoGUI for control commands, including scrolling, arrow navigation and mouse interaction. Meanwhile, YOLOv8 classifier achieved a good accuracy during training but exhibited lower stability in real-time testing, particularly with specific gestures. The findings suggest that landmark-based models provide a lightweight and efficient solution for gesture recognition into control systems to enhance user experience and lays the foundation for future work on expanding gesture vocabulary, improving its reliability under diverse conditions.

Actions (login required)