iCrawl: Improving the Freshness of Web Collections by Integrating Social Web and Focused Web Crawling

• Main Authors: Gossen, Gerhard, Demidova, Elena, Risse, Thomas
• Format: Journal Article
• Language: English
• Published: 19-12-2016
• Subjects: Computer Science - Digital Libraries; Computer Science - Information Retrieval
• DOI: 10.48550/arxiv.1612.06202

Proceedings of the 15th ACM/IEEE-CS Joint Conference on Digital
Libraries (pp. 75--84) (2015) Researchers in the Digital Humanities and journalists need to monitor,
collect and analyze fresh online content regarding current events such as the
Ebola outbreak or the Ukraine crisis on demand. However, existing focused
crawling approaches only consider topical aspects while ignoring temporal
aspects and therefore cannot achieve thematically coherent and fresh Web
collections. Especially Social Media provide a rich source of fresh content,
which is not used by state-of-the-art focused crawlers. In this paper we
address the issues of enabling the collection of fresh and relevant Web and
Social Web content for a topic of interest through seamless integration of Web
and Social Media in a novel integrated focused crawler. The crawler collects
Web and Social Media content in a single system and exploits the stream of
fresh Social Media content for guiding the crawler.