Editing Final Report 2011 (section)

===Concept===
The web crawler is the second module of the internet search application defined in the Objectives. The purpose of the web crawler module is to autonomously browse websites as explained in the [[#Background|Background Theory]] section. This web crawler module must interface with the pattern matcher module, passing web page content and accepting results to deliver to the centralised user interface.
The requirements of the web crawler included:
# Java-coded to maintain compatibility with other modules
# Accept a seed URL from which to explore
# Download pages linked in the page being explored
# Not revisit pages already visited
# Abide by ethical web crawling practices.

Ethical web crawling practices refer to not overloading servers with repeated requests and not exploring sections of internet pages the domain determines should be off-limits to web crawlers<ref name=Rodham>Rodham, Ken, Web Crawler, Brigham Young University Computer Science, http://faculty.cs.byu.edu/~rodham/cs240/crawler/index.html</ref>.