While researching how google crawls websites, I found this great piece of information on the Google Webmasters site.

I am certain google doesn't crawl every website every few seconds. Does this actually mean that when the Googlebot is crawling, it won't access the website for the duration of the crawl more than every few seconds? is this to avoid looking like a potential DoS attack? I think the wording could be clearer here!

For reference, the above image was taken from this URL: http://www.google.com/support/webmasters/bin/answer.py?answer=34439&ctx=sibling

Wednesday, April 23, 2008 2:54:15 PM (AUS Eastern Standard Time, UTC+10:00)       Comments [0]  Webmaster