The paper titled as “Multimodal Fusion Based on Information Gain for Emotion Recognition in the Wild” (E. Ghaleb, M. Popa, E. Hortal, S. Asteriadis) has been accepted at the Intelligent Systems Conference (IntelliSys 2017), which will take place in London, United Kingdom from 7 to 8 September 2017.
In this paper we present a novel approach to-wards multi-modal emotion recognition on a challenging dataset AFEW’16, composed of video clips labeled with the six basic emotions plus the neutral state. After a preprocessing stage, we employ different feature extraction techniques (CNN, DSIFT on face and facial ROI, geometric and audio based) and encoded frame-based features using Fisher vector representations. Next, we leverage the properties of each modality using different fusion schemes. Apart from the early-level fusion and the decision level fusion approaches, we propose a hierarchical decision level method based on information gain principles and we optimize its parameters using genetic algorithms. The experimental results prove the suitability of our method, as we obtain 53.06%validation accuracy, surpassing by 14% the baseline of 38.81%on a challenging dataset, suitable for emotion recognition in the wild.