A genetic programming framework to schedule webpage updates

• Published in: Information retrieval (Boston) Vol. 18; no. 1; pp. 73 - 94
• Main Authors: Santos, Aécio S. R., de Carvalho, Cristiano R., Almeida, Jussara M., de Moura, Edleno S., da Silva, Altigran S., Ziviani, Nivio
• Format: Journal Article
• Language: English
• Published: Dordrecht Springer Netherlands 01-02-2015; Springer Nature B.V
• Subjects: Computer Science; Data Mining and Knowledge Discovery; Data Structures and Information Theory; Freshness; Genetic algorithms; Image quality; Information retrieval; Information Storage and Retrieval; Internet; Machine learning; Mathematical programming; Natural Language Processing (NLP); Pattern Recognition; Performance evaluation; Performance measurement; Programming; Scheduling; Search engines; Studies; Scheduling functions; Genetic Programming; Web crawling
• DOI: 10.1007/s10791-014-9248-5
• EISSN: 1573-7659
• ISSN: 1386-4564

The quality of a Web search engine is influenced by several factors, including coverage and the freshness of the content gathered by the web crawler. Focusing particularly on freshness, one key challenge is to estimate the likelihood of a previously crawled webpage being modified. Such estimates are used to define the order in which those pages should be visited, and thus, can be exploited to reduce the cost of monitoring crawled webpages for keeping updated versions. We here present a Genetic Programming framework, called
G
P
4
C
—
Genetic Programming for Crawling
, to generate score functions that produce accurate rankings of pages regarding their probabilities of having been modified. We compare
G
P
4
C
with state-of-the-art methods using a large dataset of webpages crawled from the Brazilian Web. Our evaluation includes multiple performance metrics and several variations of our framework, built from exploring different sets of terminals and fitness functions. In particular, we evaluate
G
P
4
C
using the ChangeRate and Normalized Discounted Cumulative Gain (NDCG) metrics as both objective function and evaluation metric. We show that, in comparison with ChangeRate, NDCG has the ability of better evaluating the effectiveness of scheduling strategies, since it is able to take the
ranking
produced by the scheduling into account.