Personalized Real-Time Multimodal Mental Stress Detection using Facial Micro-Expressions and Voice Tremor Analysis with Explainable AI
Personalized Real-Time Multimodal Mental Stress Detection using Facial Micro-Expressions and Voice Tremor Analysis with Explainable AI
Authors:
Sushma Laxman Wakchaure1, Dhanashri Khairnar2,
1Department of Computer Technology, Amrutvahini Polytechnic, Sangamner
2Department of Computer Technology, Amrutvahini Polytechnic, Sangamner
Abstract - In today’s fast-moving world, mental stress has become a common problem that affects people’s health, emotions, and daily performance. Most existing stress detection methods depend on surveys, wearable devices, or manual observation, which may not always provide accurate or real-time results. To overcome these limitations, this research proposes a Personalized Real-Time Multimodal Mental Stress Detection System using Facial Micro-Expressions and Voice Tremor Analysis with Explainable Artificial Intelligence (XAI).The proposed system detects stress by analysing two important human behaviours: facial expressions and voice patterns. Subtle facial changes such as eye movement, lip tension, and facial muscle activity are captured using computer vision and deep learning techniques. At the same time, voice features like pitch changes, speech speed, tremors, jitter, and shimmer are extracted from speech signals to identify stress-related patterns. By combining both facial and voice data, the system aims to improve stress detection accuracy and reliability compared to single-mode approaches. A personalization mechanism is included to adapt the model according to each user’s normal speaking style and facial behaviour, making the system more effective for different individuals. In addition, Explainable AI methods are used to show how the system makes decisions, helping users and professionals understand the main factors responsible for stress detection. The proposed system can support applications in healthcare, workplace monitoring, online learning environments, and smart human-computer interaction systems. The main objective of this research is to develop an accurate, real-time, user-friendly, and transparent stress detection solution that can assist in early mental health awareness and prevention [1][2].
Key Words: Facial Micro-Expressions, Voice Tremor Analysis, Deep Learning, Explainable Artificial Intelligence (XAI),