Deep Convolutional Neural Networks with Transfer Learning for Visual Sentiment Analysis

• Published in: Neural processing letters Vol. 55; no. 4; pp. 5087 - 5120
• Main Authors: Usha Kingsly Devi, K., Gomathi, V.
• Format: Journal Article
• Language: English
• Published: New York Springer US 01-08-2023; Springer Nature B.V
• Subjects: Affect (Psychology); Architecture; Artificial Intelligence; Artificial neural networks; Classification; Cognition; Complex Systems; Computational Intelligence; Computer Science; Computer vision; Data augmentation; Data mining; Datasets; Deep learning; Emotions; Image classification; Image retrieval; Machine learning; Neural networks; Object recognition; Semantics; Sentiment analysis; Social networks; Visual tasks; DenseNet; Affective computing; CNN; Transfer learning; Data augmentation; Visual sentiment analysis; ResNet
• ISSN: 1370-4621; 1573-773X
• DOI: 10.1007/s11063-022-11082-3

The objective of visual sentiment analysis is to predict the positive or negative sentiment polarity evoked by images by analysing the image contents. The task of automatically recognizing sentiments in still images is inherently more challenging than other visual recognition tasks such as scene recognition, object classification, and semantic image classification since, it involves higher level of abstraction in the human cognition perspective. Sentiment classification in still images requires effective handling of large intra-class variance, scalability, subjectivity while it is also ambiguous as an image can evoke multiple sentiments. To address these issues many of the existing works focus on improving the image sentiment representation. The emergence of convolutional neural networks (CNN) has resulted in impressive performance on computer vision related tasks. The significant contribution of this work includes an exhaustive analysis on four pre-trained CNN architectural models, namely, AlexNet, GoogleNet, ResNet50, and DenseNet201 along with five data augmentation methods on five affective datasets, IAPSa, ArtPhoto, abstract paintings, MART, and EmoROI. Data augmentation is proven to provide better performance for smaller datasets. Five-fold cross validation was performed to train and evaluate the four models with data augmentation and the results demonstrate that the proposed framework is able to achieve improved performances compared to conventional techniques.