Catch-44 - Webmaster

Divide by zero

November 2008

Sun

Mon

Tue

Wed

Thu

Fri

Sat

Navigation

How to implement Media RSS

The Media RSS format is very simple, it only requires the addition of two namespaces to an RSS feed, and two elements within that namespace. The namespaces are:

xmlns:atom="http://www.w3.org/2005/Atom"
xmlns:media=http://search.yahoo.com/mrss/

The elements that need to be added are:

<media:thumbnail url="path_to_image_thumbnail" />
<media:content url="path_to_image" />

The URL's can should be absolute URL's to validate in feed validation systems, although it will still work if you use relative URL's. The thumbnail element should be used, but if you don't have a separate thumbnail image you can just point it at the same as the full size image. It would be worthwhile to use a thumbnail as you will save bandwidth and probably please your viewers with a more responsive experience.

The last thing to do is to include a link to the feed on the page where you want to display it. This points to the Media RSS feed you created earlier. The Media R

SS feed can exist anywhere in your site, just make sure you point to it in the page where you want the feed discovered.

Automatic generation of Media RSS

I have included a download of an application to generate all the files necessary to implement Media RSS on a site. This is an update to an earlier application called Focas.net.ImageGallery. This application creates a MooTools/Prototype image gallery. This update adds a Media RSS file and appropriate link to it. The download link is below. The information describing the images is extracted from the image itself. There is an earlier blog entry describing how this is done, or you can also refer to the metadata section below.

Using the application

To use the application, you need to start the application and do the following:

Choose the image directory
Note: Only images of type jpg,jpeg,gif and png will be used, and only images in the root of this folder.
Output directory
Any existing files in this folder will be overwritten. The folder will not be cleared out first if any files exist.
Title
This will be the title of the page, the heading above the images of the Image Gallery, and title of the Media RSS feed.
URL
This is optional. If you want to put the files up on a web server then this would be the path up to where you put the gallery. If you omit it, then the paths to images will be relative. This may result in feed validators claiming the feed is invalid, but it will work anyway. It is worth leaving this blank For the first attempt, so that you can see what the gallery looks like from the local file system.
Folder for images
This is the name of the folder where the actual pictures will be stored. The default if empty will be pictures.

Once you click Make the gallery! all the necessary files will be created, and automatically displayed in internet Explorer. Note that the images will not work properly in PicLens if an URL was specified as the paths are absolute URL's. The files are now ready to upload to your web server.

A note about metadata and troubleshooting

Metadata is data describing something. In this case, describing the images. This metadata is extracted from the images themselves. If using Windows Vosta then this information is easily entered via Windows Explorer or the standard picture viewer application. On previous versions of Windows, you can download the Microsoft Pro Photo tools, a free download from the Microsoft site. If you have no metadata for your images, then the application will still generate a Media RSS file, but it is better to use some metadata.

Additional resources

PicLens WebMasters guide
Microsoft Pro Photo Tools
PowerSet Media RSS links

Downloads

Source code (.NET) (89.9 KB)

Executable(140.5 KB)

Tuesday, August 12, 2008 11:18:03 PM (AUS Eastern Standard Time, UTC+10:00)

Comments [0] Downloads | JavaScript | Metadata | Programming | Webmaster

Wednesday, July 30, 2008

Moodle file upload and SCORM problems
I have been doing some work with the excellent Moodle, an open source Learning Management System, and after doing an upgrade, started experiencing problems uploading SCORM objects. The specific errors I was getting were PCLZIP_ERR_BAD_FORMAT (-10) : Unable to find End of Central Dir Record signature, and An error occurs during package unzip (sic). After a lot of searching, it seemed a few people were having similar issues, and mostly it seems Internet Explorer was being blamed. Perhaps there is also a problem with Internet Explorer, however in my case, I ended up working out where Moodle stores the files on the server and inspected the folder. The file was corrupt, I couldn't download the zip file.

After trying to manually upload the files I discovered that the disk quota had been exceeded. After doing a cleanup of the disk, everything worked well.

Sometimes when searching for the cause of a problem some facts eclipse our abilities to be a little objective about the true nature of the problem. In my case it seemed clear that the problem occurred as soon as I upgraded, when in reality the cause was my making a backup on a full server. This could have been avoided in a few ways. Firstly I should keep an eye on the server quotas, that should have been a give away. However, I think the Moodle developers should have caught an IO exception when the disk was full, and reported back to the browser. This would have saved a lot of time. Having said that, I am very impressed with Moodle, this is just one of those habits that happen a lot in programming where such potential errors as stack overflow, out of memory, or IO errors are ignored with an optimistic hope that they should never occur. With this error, I assume something worse is happening. Because the page didn't crash with an error, I can only assume the error was swallowed, and thus vital helpful information was never presented on the web page.

As soon as I have caught up with this work I will look at the code and see if I can offer a bug fix.

Update: I have also experienced problems doing a course backup, with the same root cause, not enough disk space. When doing the course backup, the messages received are An error occurred while backing up course start and An error occurred while copying the zip file to the course directory. After looking at the code, it appears the Moodle error handling only looks for 4 types of IO errors. PHP 5 has 7 types, I believe it is the type 7 error that would cause this. Unfortunately the error trapping routine has a default handler for unknown error types.

Wednesday, July 30, 2008 4:24:06 PM (AUS Eastern Standard Time, UTC+10:00)

Comments [0] Moodle | Programming | Webmaster

Thursday, June 26, 2008

Using VisiFire to create a Silverlight chart of your visitors in dasBlog

Continuing from a previous post in which I added a Google Map of the visitors to my blog, this post will discuss how to take the same information and present it in a Visifire chart. Visifire is an open source Silverlight charting component which is very easy to use and well documented.

Figure 1 - An example of the Visifire Silverlight chart tracking visitors

This post will use the same component from the previous post, the VisitorsBox.ascx, and modify the code that retrieves the location of the visitor, and add some javascript to render the chart.

First, a JavaScript array is declared which will hold the location information received by the IPHost web site:

var cities=new Array();

This array will have items pushed into it once they are retrireved:

cities.push(new Array(location, addresses[x][1]));

Once again, the code that actually calls the HTML page with the location information is covered in the earlier post.

After the Google Map is populated, the Visifire chart is prepared. To create the Visifire chart we need to do the following:

Add some HTML code into VisitorsBox.ascx
Add the Visifire.js and Visifire.xap (SilverLight binary) files to the web server
Dynamically create an XML file to feed to the Visifire.xap component
in the JavaScript, create a new Visifre object
Call the Visifire object's setDataXml method, passing the XML data in
Call the Visifire objects render method, passing in the id of the HTML layer that the chart is to be rendered to.

The HTML code is straightforward, just a layer with an id attribute:

<div id="VisifireChart"></div>

The Visifire.js can be copied wherever you want, altho the scripts folder seems an obvious candidate. The Visifre.xap file can go in the bin folder.

To create the XML, I have just concatenated a large string. Where it requires the data, I loop thru the cities array and populate the relevant information. Note this is C# code writing out a JavaScript block. Here is the header part of the XML string:

function chart() {
var chartXmlString = ''
    +'<vc:Chart xmlns:vc=\'clr-namespace:Visifire.Charts;assembly=Visifire.Charts\''
+' Width=\'500\' Height=\'300\' BorderThickness=\'0\' Theme=\'Theme3\' ';
+'LightingEnabled=\'False\' ColorSet=\'Picasso\' AnimationType=\'Type5\'>'
+'<vc:Title Text=\'Visitors by location\'/>'
+'<vc:AxisX Title=\'blog.focas.net.au\'>'
+'</vc:AxisX>'
+'<vc:AxisY Title=\'Visitors\'>'
+'</vc:AxisY><vc:DataSeries Name = \'Series1\' RenderAs=\'Column\'>';

Now, to create the data points:

for (var x=0;x<cities.length;x++) {
chartXmlString+= '<vc:DataPoint AxisLabel=\'' + cities[x][0] + '\' YValue=\'' + cities[x][1] + '\'/>';
}

And close the XML string off:

chartXmlString+='</vc:DataSeries>'
+'</vc:Chart>';

The following JavaScript code creates the chart, loads the data and renders the chart:

var vChart =new Visifire('Visifire.xap',500,300);
vChart.setDataXml(chartXmlString);
vChart.render('VisifireChart');

And thats all there is to it. As in the previous example, if you take the client side approach, then you have cross site scripting issues. So the site will need to be added to a trusted zone. If this is not suitable then you will need to run the code on the server that retrieves the locations.

Download instructions

The instructions for installing this page are in the earlier post Please remember that In order to view the example for this, you would have to add http://blog.focas.net.au to your trusted zones in Internet Explorer.

visifire.zip (11.06 KB)

Thursday, June 26, 2008 11:15:30 PM (AUS Eastern Standard Time, UTC+10:00)

Comments [3] dasBlog | Downloads | JavaScript | Webmaster | Silverlight

Tuesday, June 17, 2008

dasBlog and the Validation of viewstate MAC failed error

I have been having the following error occuring a lot in the dasBlog eventlog.

System.Web.HttpException: Validation of viewstate MAC failed. If this application is hosted by a Web Farm or cluster, ensure that <machineKey> configuration specifies the same validationKey and validation algorithm. AutoGenerate cannot be used in a cluster. ---> System.Web.UI.ViewStateException: Invalid viewstate.

As this blog is not hosted on a web farm, I searched high and low, but hadn't found anyone else reporting the same problem. After some further investigation, I think I have found the cause. It is always on the CommentView.aspx page. I tried adding comments in various browsers, and that worked without a hitch. So in frustration I did what I should have done in the first place, i.e. completely read the error message. In most instances there was an interesting user agent string such as MRSPUTNIK 1, 5, 0, 19 SW. This is mentioned in a lot of forum entries as a harvester or scraper. I suspect it is trying to post a spam message on the comments page, but because it hasn't followed the normal process, it doesn't contain the view state that the page was accepting. This is almost certainly a bot, and this is why there is no viewstate to decrypt from a previous page. I also checked the IP address, and port number. The ip addresses had many forum entries. One example was 89.149.205.199. I found an interesting site called IPillion which traces IP addresses. So adding that IP address to the url gives http://www.ipillion.com/?ip=89.149.205.199 which reports this IP address as sending lots of spam comments.

So it seems that dasBlog is sort of preventing the spam comments, altho accidentally. I hope this entry helps others having the same issue, as I couldn't find anyone using dasBlog who had the same problem, or had traced it to spammers using bots.

Tuesday, June 17, 2008 11:07:26 PM (AUS Eastern Standard Time, UTC+10:00)

Comments [6] Webmaster | dasBlog

Wednesday, June 11, 2008

Where are your blog visitors coming from in DasBlog?

See part II of this post here.

By combining the Google Maps API with an IP address host service such as HostIP.info, it is possible to present a map showing the current visitors to your blog, and to display them on a map. This implementation shows how to do this for dasBlog. The result will look something like this:

Notice that the shadow on the marker for the united States is darker, indicating more hits for that area. Each of the marker icons can be clicked to give location details and how many hits for that location.

Client side or server side location determination

The location of an IP Address can be determined at the server side, or at the client side. Each has its advantages and disadvantages. A brief but not complete list of pros and cons is:

Client side
- Pros
  - No delay in serving the page as the processing is done after the page has been serves to the client.
  - Easy to implement
- Cons
  - Requires a call to a web service on a different domain which will cause security concerns
  - Does not allow for users to be able to modify what is their perceived location
Server side
- Pros
  - More control as you are not dependent on an external web service
  - It is easier to cache results if required
- Cons
  - More difficult to implement, especially if you want to keep to the architecture of the software you are developing for
  - You must maintain and update your own databse of IP address ranges and locations

The client side solution

This solution relies on a client side solution, taking advantage of the excellent Prototype JavaScript Framework. Further down the track I might possibly change this to a server side solution, but as it is only for an administrative page it is not so important. Note that there are security concerns with doing this client side. It is necessary top make an AJAX call to an external webservice, and this could be malicious, especially if a site were hacked. In order to view the example for this, you would have to add http://blog.focas.net.au to your trusted zones in Internet Explorer. If you do this, please make sure you remove it immediately afterwards. Usually I would only add a trusted site if I were using it frequently. The example page can be viewed at Visitors.aspx (Please scroll down to the bottom of the page as there are styling issues. The page may need refreshing)

I created a server page called Visitors.aspx, and a user control called VisitorsBox.ascx. These are directly based upon Referrers.aspx and ReferrersBox.ascx in the standard DasBlog source code.

How it works

C# code in the page creates JavaScript which is then registered with the RegisterClientScriptBlock method of the ClientScript object. The javascript requires some dynamic variables to be created and this is why it is in the page. Also the way dasBlog is set-up seems to make it difficult to add JavaScript to just one page without using this method. Atwo dimensional array is created which contains unique IP addresses and a count of hits for those addresses. THis is then processed on the client side The JavaScript code to create the map follows, it is well documented in the Google Maps API documentation, so i won't say too much about it.

if (GBrowserIsCompatible()) {
     map = new GMap2(document.getElementById('map_canvas'));
     map.setCenter(new GLatLng(37.4419, -122.1419), 1);
     var mapTypeControl = new GMapTypeControl();
     var topRight = new GControlPosition(G_ANCHOR_TOP_RIGHT, new GSize(10,10));
     var bottomRight = new GControlPosition(G_ANCHOR_BOTTOM_RIGHT, new GSize(10,10));
     map.addControl(mapTypeControl, topRight);
     map.addControl(new GSmallMapControl());

Now we need to loop thru the IP address array and call the hostip.info page to get the city/state/country information. There are different calls that can be made to return HTML or XML. I used the HTML version as it is a smaller response, and easy to parse with regular expressions. If there is no information for an IP address it will return (Unknown). Thed regular expression trims them out. This database seems to be maintained by one person, so consider donating to them.

var myAjax;
      
 for (var x=0;x<addresses.length;x++) {
     myAjax = new Ajax.Request('http://api.hostip.info/get_html.php?ip=' + addresses[x][0] , 
     {
         method: 'get', 
         onSuccess: function(originalRequest) {
            var result=originalRequest.responseText;
            var tokens=result.split('\n');
            var count=addresses[x][1];
            var country=/\:[^\(]+/i.exec(tokens[0])[0].substring(1).strip();
            var city=/\:[^\(]+/i.exec(tokens[1])[0].substring(1).strip();
            var location=(city.length>0)? city + ',':'';
             location+=country;     
            varmessage=location + ': ' + addresses[x][1] + ' visitor';
            if (addresses[x][1] > 1) message+='s';

Now we can call the GClientGeocoder object to turn the city/state/country information into a latitude/longtitude combination. The GClientGeocoder's getLatLng object expects a method to be passed in to handle a callback. In this code, it is used to create the marker on the map which can be clicked to view more information.

   var geocoder = new GClientGeocoder();
   geocoder.getLatLng(location,
   function(point) {  
      if (point) { 
         var marker = new GMarker(point);   
         map.addOverlay(marker);   
         GEvent.addListener(marker, 'click', function() {
         marker.openInfoWindow(message);
      });
   }
   });

This javascript is created in the C# code using a StringBuilder object, and is then added to the page using the RegisterClientScriptBlock method of the ClientScript object.

Page.ClientScript.RegisterClientScriptBlock(this.GetType(), "ipaddressdata", sb.ToString(), true);

Download

The download code contains the Visitors.aspx and Visitors.ascx page and component. These should be dropped into the newtelligence.DasBlog.Web folder of your dasBlog source code. Before going any further you will need to change the Google Maps key in VisitorsBox.ascx.cs file.

Page.ClientScript.RegisterClientScriptBlock(this.GetType(), "googlemaps",
    @"<script src='http://maps.google.com/maps?file=api&amp;v=2&amp;key=YOUR KEY GOES HERE'"+" type='text/javascript'></script>");

When you have compiled the code, drop those two files into the root of your website, and the newtelligence.DasBlog.Web.dll into the bin folder of your website. Remember to change the trust settings for your blog, and enjoy seeing who is visiting your site. Note that the Google Maps JavaScript and Prototype JavaScripts are served up from the Google servers so you don't need to download them.

Visitors.zip (10.57 KB)

Wednesday, June 11, 2008 11:53:11 PM (AUS Eastern Standard Time, UTC+10:00)

Comments [0] Downloads | JavaScript | Webmaster

Tuesday, June 03, 2008

Web log file parsing with c#

In the past week I have been doing a lot of analysis on the web analytics reports that are being generated by AWStats (A good Open Source product, don't be put off by the arrogant download Firefox message) and another commercial product that we have an Enterprise license for. I won't name this product, as I am very disillusioned with it, its accuracy is very questionable.

In order to determine how accurate the statistics were, i chose one days log file for a reasonably busy website. This site is mainly a Monday to Friday website with peaks around 10 AM to 4 PM, and another small peak 6 PM to 8PM. Weekends see a lower usage, so I used a sunday as this would be the smallest log I could reasonably analyse. Thee log contained just over 100,000 entries. In order to analyse it, I used Visual Studion 2008, filtering using regular expressions. After a while I realised I would like to review these stats on an ongoing basis, so i wrote a small log parser. There is an excellent free parser available from Microsoft called Log Parser 2.2 however I wanted direct control, because I also want to create some graphs. My first version creates a beautiful graph using the excellent Open Source Silverlight data visualisation component called Visifire. I haven't included that code in this download as i wanted to clean it up and make it more re-usable and extensible.

This tool is not particularly robust as it is just a throwaway utility, but I tried to make it a bit extensible for future requirments. The log files I am parsing are Apache log files. So I designed a simple ILogFormatReader interface, and created an ApacheLogParser implementation. This doesn't populate all the fields, but it's easy to see how it works to finish off the implementation if more information is necessary.

The main issue with parsing log files is how they are separated. in this case the log file is space separated. Any fields that have spaces in them are surrounded by double quotes or square brackets. Another consideration is what to do if a log entry comes in with an invalid format. I didn't worry about this too much, as the web logs should work well if the first line is correct. if this code was to go into production then of course that would have to be refactored.

Not all log file are created equal

Log parser example output Each web server can be customised to record different informatin in the log files. Additionally proxy servers can modify what is sent thru to the web server, so it is important to check the format of any logfile before parsing. The log files I am parsing are using an Apache log file format. More information can be found on the Apache log files page and the Microsoft Log file formats in IIS (6.0) page.

For those interested, I found the statistics in AWStats very close to what I believe they should be. There were inaccuracies that I couldn't explain. To be fair, we are running an old version of AWStats so I assume that this might have been addressed in a newer version. The very expensive commercial application we use reports approximately twice the bandwidth that it should, and does not understand how to handle the JSESSIONID tacked on to the end of some JSP applications. It confuses the real resource with the session ID, and we get very iaccurate statistics as a result.

There is some minimalistic reporting to give an idea of how to use the parser. The output looks like thie image here.

There is a download included below. I would be interested to hear if anyone finds this useful. I hope to update this application at some stage to include some graphing output. I might use Visifire as mentioned earlier, or possibly Microsoft Excel. I think that is a better option as it would allow for richer maniupulation of the reporting after the log files have been parsed. Excel has such powerful features it would require good justification not to use it as a reporting mechanism. I think that AWStats doesn't do their reports justice by presenting them as they do, Everyones reporting requirements are different.

Download

The applicaition can be downloaded here:

LogParsing.zip (7 KB)

Tuesday, June 03, 2008 11:33:35 PM (AUS Eastern Standard Time, UTC+10:00)

Comments [0] Downloads | Webmaster

Tuesday, May 27, 2008

How to add a provider to the Internet Explorer search bar

Internet Explorer 7 and above has a search bar with a default provider of live search. It also contains a link to other search providers such as Microsoft and Wikipedia. It is very easy to add a new provider to this search bar. It can be done manually, but this article describes how to change a web page to enable auto-detection of a search provider, plus how to create a link to that provider. Firefox 2 and above also has a search sidebar that supports this feature.

The search provider requires an XML file which adheres to the OpenSearch Description schema. It is easy enough to create this, but if you want to use Internet Explorer to create it for you then you can use the Find more providers option in the search bar dropdown. On the page it displays you place the search page for your site with the word TEST instead of a valid search keyword. This will only work for sites that use GET rather than POST for search queries. Once you have entered the URL, there is a View XML option which will display the necessary XML. It will look like this:

<? xml version = "1.0" encoding = "UTF-8" ?>
<OpenSearchDescription xmlns = "http://a9.com/-/spec/opensearch/1.1/" >
<ShortName>blog.focas.net.au</ShortName>
<Description>blog.focas.net.au provider</Description>
<InputEncoding>UTF-8</InputEncoding>
<Url type="text/html"
template="http://blog.focas.net.au/SearchView.aspx?q={searchTerms}"/>
</OpenSearchDescription>

The ShortName will show up in the search bar. This file should be saved somewhere on your website. it doesn't matter where as it is the metadata in the pages that will point to its location.

On any page where you want Internet Explorer to automatically detect the search provider, you need to add some metadata into the head section of the page. This will look like this:

This tells Internet Explorer that there is a search provider from the rel attribute containing the value search. The type is a mime type referring to the Open Search Description xml format. the href is the location of the XML file you saved earlier.

Once this is in the page, when you browse to this page, Internet explorer will change the drop-down icon colour on the search toolbar to orange. Like this:

Figure 1: Search provider glowing when it has discovered a search provider

By clicking on the drop-down, the provider will be displayed. It has not been installed at this stage, but is available whenever the page is viewed:

Figure 2: Search provider displayed in the search provider drop-down

To allow the user of a page to install the search provider in Firefox2+ or Internet Explorer 7+, you need to create a link on a page that calls some javascript. The window.external method must be called in order for the link to work. The following script is a modified version of a script on the Mozilla developer center

function installSearchEngine(url) {
if (window.external && ("AddSearchProvider" in window.external)) {
    // Firefox 2 and IE 7, OpenSearch
    window.external.AddSearchProvider(url);
} else {
    // No search engine support (IE 6, Opera, etc).
    alert("Sorry, your browser doesn't provide search engine support");
}
}

This page has a search provider in its metadata, so you can see the effect in the search box if using IE7+. Alternatively you can try installing the search engine by clicking here.

Tuesday, May 27, 2008 10:26:21 PM (AUS Eastern Standard Time, UTC+10:00)

Comments [0] Metadata | Webmaster

Saturday, May 17, 2008

Image Gallery using metadata

Image gallery using metadata

This is a small application that takes all the images in a directory and creates a lightbox style AJAX image gallery that is web ready. It reads the metadata in the picture to extract the title, description, keywords and rating. Primarily I wrote this to experiment with some C# 3 features, such as LINQ.

There are no options in the program, its fairly simple, point it at an input directory, an output directory and click Make the gallery! And it does its stuff. Its not overly sophisticated, it doesn’t generate thumbnails, just uses width and height attributes on the img tags. All the necessary support files will be copied across to the output directory. A page called index.html is generated and automatically displayed when complete.

Metadata collection

Metadata is collected using the System.Windows.Media.Imaging namespace. This is part of the Windows Presentation Foundation. When I tested this it worked well on Windows Vista, but when I tested it on a machine running Windows XP SP2, I got a codec not available error when accessing the metadata. I got around this by installing Microsoft Photo Info which is a fantastic utility for XP that incorporates read/write access to image metadata, It has explorer integration, and I highly recommend it if you like adding metadata to your images. It can also be very helpful for those who have upgraded from Windows Vista to Windows XP.

The code to access the metadata is straightforward:

using(Stream stream = fii.OpenRead()) {

BitmapDecoder decoder = BitmapDecoder.Create(stream, BitmapCreateOptions.None, BitmapCacheOption.Default);

BitmapFrame frame = decoder.Frames[0];

BitmapMetadata metadata = (BitmapMetadata)frame.Metadata;

caption = metadata.Title;

if(metadata.Subject != null) {

caption += " - " + metadata.Subject;

}

I tried to close the stream, hoping that the metadata would be cached, but I suspect it uses Lazy loading because it would throw an error as soon as I accessed the metadata.

LINQ seemed a good idea for filtering the files. There may be a better way but this worked just fine.

string[] files = Directory.GetFiles(directoryToProcess);

var query = from f in files

where (new string[] { ".jpg", ".png", ".gif" , ".jpeg"}).Contains(Path.GetExtension(f).ToLower())

select f;

return query;

Having the list of file extensions as the first part of the where clause didn't please me, but that is just aesthetics.

Javascript/CSS

The lightbox effect is achieved by using the MooTools JavaScript framework, and another library and example from phatfusion which creates the lightbox. This program just encapsulates the HTML generation, metadata extraction and file copying.

Scope for improvement

There is a lot of scope for improvement. The class layout is fairly simple. Interfaces could be added, and a plug-in approach to allow for different LightBox or similar effects. As this was more of an experiment than a robust utility I didn’t get too precious about such design considerations. I hope it helps a few people to make a gallery for themselves.

Focas.NET.ImageGallery.zip (84.1 KB)
Saturday, May 17, 2008 10:11:52 PM (AUS Eastern Standard Time, UTC+10:00)

Comments [0] Downloads | Metadata | Webmaster

Wednesday, April 23, 2008

Googlebot frequency

While researching how google crawls websites, I found this great piece of information on the Google Webmasters site.

I am certain google doesn't crawl every website every few seconds. Does this actually mean that when the Googlebot is crawling, it won't access the website for the duration of the crawl more than every few seconds? is this to avoid looking like a potential DoS attack? I think the wording could be clearer here!

For reference, the above image was taken from this URL: http://www.google.com/support/webmasters/bin/answer.py?answer=34439&ctx=sibling

Wednesday, April 23, 2008 2:54:15 PM (AUS Eastern Standard Time, UTC+10:00)

Comments [0] Webmaster