Supermarket shopping assistant for the visually impairs

Cheng, Puei Ying (2025) Supermarket shopping assistant for the visually impairs. Final Year Project, UTAR.

PDF Download (5Mb)

Abstract

This project introduces a Supermarket Shopping Assistant designed to support Visually Impaired (VI) individuals in performing grocery shopping independently. Shopping in supermarkets presents major challenges for VI users, as identifying products, locating items on shelves, and reading packaging information typically require external assistance. To address this gap, the proposed system integrates computer vision, optical character recognition, and text-to-speech technologies into an Android mobile application that provides real-time product recognition, hand guidance, and product information retrieval. The system architecture was developed around two complementary approaches for product identification: a product detection model using Roboflow API, and a text detection approach leveraging Google Cloud Vision API. While the detection model achieved high accuracy in recognizing standard grocery products, the text detection approach provided greater robustness by always generating an output, even when object detection failed. In addition, hand detection was implemented using MediaPipe to guide the user’s hand toward the detected product through audio feedback. Once the item was acquired, the user could capture its packaging details, which were processed using OCR and summarized by the Gemini API before being read aloud, ensuring clarity and conciseness. Evaluation results demonstrated that the system successfully fulfilled its objectives, with each module performing reliably under controlled conditions. Although challenges such as dependence on cloud services were noted, the system still provided a practical and accessible solution. By enabling VI individuals to identify, reach, and evaluate grocery products independently, this project contributes to enhancing daily autonomy and highlights the potential of AI-driven assistive technologies to promote inclusivity in society.

Actions (login required)