Images2Poem in different contexts with Dual‐CharRNN

• Published in: CAAI Transactions on Intelligence Technology Vol. 7; no. 4; pp. 685 - 694
• Main Authors: Yan, Jie, Xie, Yuxiang, Luan, Xidao
• Format: Journal Article
• Language: English
• Published: Beijing John Wiley & Sons, Inc 01-12-2022; Wiley
• Subjects: Aesthetics; Artificial intelligence; Classification; computer vision; Datasets; deep learning; Emotions; Human performance; image captioning; Image classification; Image enhancement; Keywords; Language; Machine translation; Methods; Natural language; Neural networks; Poetry; poetry generation; Poets; Readability; Semantics
• ISSN: 2468-2322; 2468-2322
• DOI: 10.1049/cit2.12089

Image to caption has attracted extensive research attention recently. However, image to poetry, especially Chinese classical poetry, is much more challenging. Previous works mainly focus on generating coherent poetry without taking the contexts of poetry into account. In this paper, we propose an Images2Poem with the Dual‐CharRNN model which exploits images to generate Chinese classical poems in different contexts. Specifically, we first extract a few keywords representing elements from the given image based on multi‐label image classification. Then, these keywords are expanded to related ones with the planning‐based model. Finally, we employ Dual‐CharRNN to generate Chinese classical poetry in different contexts. A comprehensive evaluation of human judgements demonstrates that our model achieves promising performance and is effective in enhancing poetry's semantic consistency, readability, and aesthetics. We present an Images2Poem with the Dual‐CharRNN model exploiting images to generate Chinese classical poems in different contexts, which effectively improves the semantic consistency, readability and aesthetics of the generated poetry.