Editing Final Report 2012 (section)

====Pre-Indexed Web Data====
The internet is a massive place, and the time it would take to crawl every single web page - even by the WebCrawler developed by last year's group - would be prohibitively long. Therefore it was proposed that we search for a means to traverse the web faster. With search engines such as Google regularly trawling the internet and caching the data, it was suggested we find a means to access this pre-indexed data so that search times could be significantly reduced (think how quickly Google returns its search results!). However, the data collected by search engines is not publicly available, so an alternative approach was required. We looked into the possibility of using a service such as Amazon's Elastic Cloud Compute (EC2) to speed the process up. Further investigation indicated that this would require a substantial re-writing of the WebCrawler - which is designed to use a graphical user interface for its input - and only side-stepped the issue. Amazon EC2, and other similar cloud computing solutions, would only provide a means to run large numbers of instances of the WebCrawler simultaneously and, given enough instances, this would be harsh on popular websites with large numbers of links to them, which would be receiving requests from many instances in a short space of time, potentially (accidentally!) causing a Distributed Denial of Service attack as the crawlers use up all the servers' bandwidth.

Given the amount of time and effort required to fully evaluate the possibilities for speeding up trawling the internet, and the reworking of the WebCrawler code, this would make an interesting and useful project in itself for a future group to undertake.