Emotion Detection AI Using Webcam and Voice Analysis

Prashika Dabhade, Ganesh Dongre, P. R. Wankhede, Devendra Bhuyar
Department of Electronics and Telecommunicatios Engineering,

CSMSS Chh. Shahu College of Engineering, Chhatrapati Sambhajinagar - 431002, Maharashtra, India.

prashikadabhade7@gmail.com, pravin.india1@gmail.com, devbhuyar@gmail.com, ganeshbdongre@gmail.com

Abstract— This paper presents the design and implementation of an Emotion Detection AI system that identifies human emotional states in real time using two independent sensing modalities: facial expression captured through a webcam and vocal characteristics captured through a microphone. The system is implemented as a full-stack web application comprising a React.js frontend and a FastAPI backend. Facial emotion recognition is performed using OpenCV Haar Cascade classifiers for face, eye, and smile detection, combined with a rule-based scoring engine that analyses brightness, contrast, and structural cues extracted from the detected facial region. Voice emotion recognition is performed by extracting Mel-Frequency Cepstral Coefficients, Root Mean Square energy, pitch, and zero-crossing rate from short audio recordings using the Librosa signal-processing library, which are subsequently classified using a second rule-based engine. Both pipelines independently classify the input into one of seven standard emotion categories — Happy, Sad, Angry, Fear, Surprise, Disgust, and Neutral — and return a confidence score alongside the predicted label. Detected emotions and session metadata are persisted in a SQLite database, enabling later review through a dedicated history interface. Experimental evaluation on a self-collected test set of posed facial expressions and voice recordings shows an overall facial emotion classification accuracy of 70.0% and a voice emotion classification accuracy of 67.1%, with end-to-end detection latency under 700 milliseconds for both modalities. The results demonstrate that a lightweight, dependency-light architecture — avoiding heavyweight deep learning frameworks — can deliver a functional, real-time, dual-modality emotion recognition system suitable for deployment on standard consumer hardware without GPU acceleration.

Keywords— Emotion Detection, Facial Expression Recognition, Speech Emotion Recognition, OpenCV, Haar Cascade, Librosa, Human-Computer Interaction, Affective Computing, FastAPI, React.js.