A Multimodal Emotion-Aware Product Recommendation System Integrating Real-Time Facial Expression Recognition and BERT-Based Sentiment Analysis

Abstract

This study presents a multimodal emotion-aware product recommendation system that integrates real-time Facial Expression Recognition (FER) and transformer-based sentiment analysis to enhance personalization in digital environments. Traditional recommender systems rely mainly on historical interactions or textual reviews and often overlook users’ current emotional states, leading to inappropriate recommendations. To address this limitation, the proposed system fuses emotional cues from facial expressions and usergenerated text. FER is performed across seven emotion categories—happy, sad, angry, fear, disgust, surprise, and neutral—using a fine-tuned EfficientNetB0 model trained on JAFFE, CK++, FER subsets, and selfcollected webcam images, achieving an overall accuracy of 88% in realtime conditions. Sentiment analysis uses a fine-tuned DistilBERT model that classifies text into positive, neutral, and negative categories with accuracy exceeding 90%. A rule-based multimodal fusion strategy combines outputs from both modalities, resolving conflicting emotional cues and improving emotional inference reliability by approximately 10–12% compared to unimodal approaches. The inferred emotional state is mapped to a structured recommendation database, generating personalized product suggestions. The system is implemented using a Streamlit-based interface. Experimental results indicate that the multimodal approach produces recommendations that are better than those of single-modality systems.