A Deep Learning Perspective on Beauty, Sentiment, and Remembrance of Art

• Published in: IEEE access Vol. 7; pp. 73694 - 73710
• Main Authors: Cetinic, Eva, Lipic, Tomislav, Grgic, Sonja
• Format: Journal Article
• Language: English
• Published: Piscataway IEEE 2019; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: Aesthetics; Artificial intelligence; Artificial neural networks; Artists; Color; Computational modeling; Computer vision; Convolutional neural networks; Decision analysis; Decision trees; Deep learning; Feature extraction; fine art; image aesthetics; Image classification; image memorability; Machine learning; Object recognition; Task analysis; visual sentiment; Visualization
• ISSN: 2169-3536; 2169-3536
• DOI: 10.1109/ACCESS.2019.2921101

With the emergence of large digitized fine art collections and the successful performance of deep learning techniques, new research prospects unfold in the intersection of artificial intelligence and art. In order to explore the applicability of deep learning techniques in understanding art images beyond object recognition and classification, we employ convolutional neural networks (CNN) to predict scores related to three subjective aspects of human perception: aesthetic evaluation of the image, sentiment evoked by the image, and memorability of the image. For each concept, we evaluate several different CNN models trained on various natural image datasets and select the best performing model based on the qualitative results and the comparison with existing subjective ratings of artworks. Furthermore, we employ different decision tree-based machine learning models to analyze the relative importance of various image features related to the content, composition, and color in determining image aesthetics, visual sentiment, and memorability scores. Our findings suggest that content and image lighting have significant influence on aesthetics, in which color vividness and harmony strongly influence sentiment prediction, while object emphasis has a high impact on memorability. In addition, we explore the predicted aesthetic, sentiment, and memorability scores in the context of art history by analyzing their distribution in regard to different artistic styles, genres, artists, and centuries. The presented approach enables new ways of exploring fine art collections based on highly subjective aspects of art, as well as represents one step forward toward bridging the gap between traditional formal analysis and the computational analysis of fine art.