A Novel Web Scraping Approach Using the Additional Information Obtained From Web Pages

• Published in: IEEE access Vol. 8; pp. 61726 - 61740
• Main Author: Uzun, Erdinc
• Format: Journal Article
• Language: English
• Published: Piscataway IEEE 2020; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: algorithm design and analysis; Computational efficiency; Data collection; Data mining; Data structures; document object model; Feature extraction; Monitoring; Parsers; Prediction algorithms; Searching; Strings; Task analysis; Text processing; web crawling and scraping; Web pages; Websites
• ISSN: 2169-3536; 2169-3536
• DOI: 10.1109/ACCESS.2020.2984503

Web scraping is a process of extracting valuable and interesting text information from web pages. Most of the current studies targeting this task are mostly about automated web data extraction. In the extraction process, these studies first create a DOM tree and then access the necessary data through this tree. The construction process of this tree increases the time cost depending on the data structure of the DOM Tree. In the current web scraping literature, it is observed that time efficiency is ignored. This study proposes a novel approach, namely UzunExt, which extracts content quickly using the string methods and additional information without creating a DOM Tree. The string methods consist of the following consecutive steps: searching for a given pattern, then calculating the number of closing HTML elements for this pattern, and finally extracting content for the pattern. In the crawling process, our approach collects the additional information, including the starting position for enhancing the searching process, the number of inner tag for improving the extraction process, and tag repetition for terminating the extraction process. The string methods of this novel approach are about 60 times faster than extracting with the DOM-based method. Moreover, using these additional information improves extraction time by 2.35 times compared to using only the string methods. Furthermore, this approach can easily be adapted to other DOM-based studies/parsers in this task to enhance their time efficiencies.