Neural Transfer Learning For Vietnamese Sentiment Analysis Using Pre-trained Contextual Language Models

• Published in: 2021 IEEE International Conference on Machine Learning and Applied Network Technologies (ICMLANT) pp. 1 - 5
• Main Authors: Le, An Pha, Vu Pham, Tran, Le, Thanh-Van, Huynh, Duy V.
• Format: Conference Proceeding
• Language: English
• Published: IEEE 16-12-2021
• Subjects: Analytical models; Bit error rate; Conferences; Context modeling; Neural Transfer Learning; Pre-trained Language Model; Sentiment analysis; Task analysis; Transfer learning; Vietnamese Sentiment Analysis
• DOI: 10.1109/ICMLANT53170.2021.9690554

We propose a fine-tuning methodology and a comprehensive comparison between state-of-the-art pre-trained language models (PLM) when applying to Vietnamese Sentiment Analysis. The fine-tuning architecture includes three main components: (1) pre-processing, (2) a pre- trained language model, and (3) a multi-layer perceptron. The method exploits pre-trained contextual language models in order to represent input sentences. Pre-trained contextual language models are belong to three different kinds: multilingual, cross-lingual and monolingual. We conduct experiments to evaluate trained classifiers fine-tuned using five different contextual language models. The experimental results on two open-access datasets show that the sentiment classifiers trained using the monolingual language model outperform of which cross-lingual and monolingual language models. The results provide an additional evidence about the representation power of monolingual PhoBERT in comparison with multilingual BERT and cross-lingual XLM.