A novel algorithm for extracting the user reviews from web pages

• Published in: Journal of information science Vol. 43; no. 5; pp. 696 - 712
• Main Authors: Uçar, Erdem, Uzun, Erdinç, Tüfekci, Pınar
• Format: Journal Article
• Language: English
• Published: London, England SAGE Publications 01-10-2017; Bowker-Saur Ltd
• Subjects: Algorithms; Artificial intelligence; Data mining; Feature extraction; Hypertext; Machine learning; Recommender systems; Sentiment analysis; Tags; Websites; Efficient extraction; web data extraction; web user reviews
• ISSN: 0165-5515; 1741-6485
• DOI: 10.1177/0165551516666446

Extracting the user reviews in websites such as forums, blogs, newspapers, commerce, trips, etc. is crucial for text processing applications (e.g. sentiment analysis, trend detection/monitoring and recommendation systems) which are needed to deal with structured data. Traditional algorithms have three processes consisting of Document Object Model (DOM) tree creation, extraction of features obtained from this tree and machine learning. However, these algorithms increase time complexity of extraction process. This study proposes a novel algorithm that involves two complementary stages. The first stage determines which HTML tags correspond to review layout for a web domain by using the DOM tree as well as its features and decision tree learning. The second stage extracts review layout for web pages in a web domain using the found tags obtained from the first stage. This stage is more time-efficient, being approximately 21 times faster compared to the first stage. Moreover, it achieves a relatively high accuracy of 96.67% in our experiments of review block extraction.