Prediction of Head Movement in 360-Degree Videos Using Attention Model

• Published in: Sensors (Basel, Switzerland) Vol. 21; no. 11; p. 3678
• Main Authors: Lee, Dongwon, Choi, Minji, Lee, Joohyun
• Format: Journal Article
• Language: English
• Published: Basel MDPI AG 25-05-2021; MDPI
• Subjects: Algorithms; attention model; Augmented reality; Autoregressive processes; Communication; Deep learning; Distance learning; GRU; Head movement; LSTM; Machine learning; Multilayers; Neural networks; Prediction models; Recurrent neural networks; Regression analysis; Root-mean-square errors; Support vector machines; time-series prediction; Traffic flow; Video transmission; Virtual communities; Virtual reality; Vision
• ISSN: 1424-8220; 1424-8220
• DOI: 10.3390/s21113678

In this paper, we propose a prediction algorithm, the combination of Long Short-Term Memory (LSTM) and attention model, based on machine learning models to predict the vision coordinates when watching 360-degree videos in a Virtual Reality (VR) or Augmented Reality (AR) system. Predicting the vision coordinates while video streaming is important when the network condition is degraded. However, the traditional prediction models such as Moving Average (MA) and Autoregression Moving Average (ARMA) are linear so they cannot consider the nonlinear relationship. Therefore, machine learning models based on deep learning are recently used for nonlinear predictions. We use the Long Short-Term Memory (LSTM) and Gated Recurrent Unit (GRU) neural network methods, originated in Recurrent Neural Networks (RNN), and predict the head position in the 360-degree videos. Therefore, we adopt the attention model to LSTM to make more accurate results. We also compare the performance of the proposed model with the other machine learning models such as Multi-Layer Perceptron (MLP) and RNN using the root mean squared error (RMSE) of predicted and real coordinates. We demonstrate that our model can predict the vision coordinates more accurately than the other models in various videos.