Emotion recognition companion robot

Lau, Tin Tin (2025) Emotion recognition companion robot. Final Year Project, UTAR.

Preview

PDF Download (3561Kb) | Preview

Abstract

Existing companion robots often lack the depth to recognize a broad range of human emotions or engage in context-aware conversations. These limitations reduce their effectiveness in providing genuine companionship and emotional support. This project introduces Wall-E which is an intelligent companion robot designed to bridge this gap by combining computer vision and large language models to interpret user emotions and generate empathetic responses. This robot is composed of emotion detection module and the conversational response module. Each of the module is implemented as independent ROS2 nodes to ensure modularity and asynchronous communication. The emotion detection module employs a CNN model to analyse facial expressions captured in real time. Detected emotions are published through ROS2 topics for use by other nodes. Simultaneously, the conversational response module enables voice-based interaction. It captures user speech using a microphone, transcribes it with the Whisper API and combines the resulting text with the detected emotion. These inputs are passed to an OpenAI GPT agent to generate emotionally relevant responses. The replies are then converted to speech using the gTTS engine and played through the robot’s speaker. The integration of ROS2 allows seamless communication between modules using a publish-subscribe architecture. A structured verification plan was implemented to ensure system reliability including testing of the CNN model for emotion classification, audio transcription accuracy, context-aware response generation, and speech synthesis clarity.

Actions (login required)