Media RSS (MRSS) and PicLens support in a web page

Media RSS (MRSS) is an XML format which extends the RSS format. By using this format, a website can be enabled to provide rich media functionality. This can be used for images, audio and video. This entry discusses enabling Media RSS for images. Video is very easy to implement, but will not be discussed here. By using PicLens in Firefox, the result looks like below. It is very fast, full screen, and has nice facilities to scroll around the images, zoom in and out and other such things. Here I will discuss a simple implementation, and you can scroll below to download a small .NET application to automate this process. If you already have a Media RSS viewing application, then check out an example of the output (Note, copyright links were added after the page was generated).

How to implement Media RSS

The Media RSS format is very simple, it only requires the addition of two namespaces to an RSS feed, and two elements within that namespace. The namespaces are:

xmlns:atom="http://www.w3.org/2005/Atom"
xmlns:media=http://search.yahoo.com/mrss/

The elements that need to be added are:

<media:thumbnail url="path_to_image_thumbnail" />
<media:content url="path_to_image" />

The URL's can should be absolute URL's to validate in feed validation systems, although it will still work if you use relative URL's. The thumbnail element should be used, but if you don't have a separate thumbnail image you can just point it at the same as the full size image. It would be worthwhile to use a thumbnail as you will save bandwidth and probably please your viewers with a more responsive experience.

The last thing to do is to include a link to the feed on the page where you want to display it. This points to the Media RSS feed you created earlier. The Media R

SS feed can exist anywhere in your site, just make sure you point to it in the page where you want the feed discovered.

<link rel="alternate" href="imagegallery\media.xml" type="application/rss+xml" />

Automatic generation of Media RSS

I have included a download of an application to generate all the files necessary to implement Media RSS on a site. This is an update to an earlier application called Focas.net.ImageGallery. This application creates a MooTools/Prototype image gallery. This update adds a Media RSS file and appropriate link to it. The download link is below. The information describing the images is extracted from the image itself. There is an earlier blog entry describing how this is done, or you can also refer to the metadata section below.

Using the application

To use the application, you need to start the application and do the following:

• Choose the image directory
  Note: Only images of type jpg,jpeg,gif and png will be used, and only images in the root of this folder.
• Output directory
  Any existing files in this folder will be overwritten. The folder will not be cleared out first if any files exist.
• Title
  This will be the title of the page, the heading above the images of the Image Gallery, and title of the Media RSS feed.
• URL
  This is optional. If you want to put the files up on a web server then this would be the path up to where you put the gallery. If you omit it, then the paths to images will be relative. This may result in feed validators claiming the feed is invalid, but it will work anyway. It is worth leaving this blank For the first attempt, so that you can see what the gallery looks like from the local file system.
• Folder for images
  This is the name of the folder where the actual pictures will be stored. The default if empty will be pictures.

Once you click Make the gallery! all the necessary files will be created, and automatically displayed in internet Explorer. Note that the images will not work properly in PicLens if an URL was specified as the paths are absolute URL's. The files are now ready to upload to your web server.

A note about metadata and troubleshooting

Metadata is data describing something. In this case, describing the images. This metadata is extracted from the images themselves. If using Windows Vosta then this information is easily entered via Windows Explorer or the standard picture viewer application. On previous versions of Windows, you can download the Microsoft Pro Photo tools, a free download from the Microsoft site. If you have no metadata for your images, then the application will still generate a Media RSS file, but it is better to use some metadata.

Additional resources

PicLens WebMasters guide
Microsoft Pro Photo Tools
PowerSet Media RSS links

Downloads

Source code (.NET) (89.9 KB)

Executable(140.5 KB)

Tuesday, August 12, 2008 11:18:03 PM (AUS Eastern Standard Time, UTC+10:00)       Comments [0]  Downloads | JavaScript | Metadata | Programming | Webmaster

Moodle file upload and SCORM problems
I have been doing some work with the excellent Moodle, an open source Learning Management System, and after doing an upgrade, started experiencing problems uploading SCORM objects. The specific errors I was getting were PCLZIP_ERR_BAD_FORMAT (-10) : Unable to find End of Central Dir Record signature, and An error occurs during package unzip (sic). After a lot of searching, it seemed a few people were having similar issues, and mostly it seems Internet Explorer was being blamed. Perhaps there is also a problem with Internet Explorer, however in my case, I ended up working out where Moodle stores the files on the server and inspected the folder. The file was corrupt, I couldn't download the zip file.

After trying to manually upload the files I discovered that the disk quota had been exceeded. After doing a cleanup of the disk, everything worked well.

Sometimes when searching for the cause of a problem some facts eclipse our abilities to be a little objective about the true nature of the problem. In my case it seemed clear that the problem occurred as soon as I upgraded, when in reality the cause was my making a backup on a full server. This could have been avoided in a few ways. Firstly I should keep an eye on the server quotas, that should have been a give away. However, I think the Moodle developers should have caught an IO exception when the disk was full, and reported back to the browser. This would have saved a lot of time. Having said that, I am very impressed with Moodle, this is just one of those habits that happen a lot in programming where such potential errors as stack overflow, out of memory, or IO errors are ignored with an optimistic hope that they should never occur. With this error, I assume something worse is happening. Because the page didn't crash with an error, I can only assume the error was swallowed, and thus vital helpful information was never presented on the web page.

As soon as I have caught up with this work I will look at the code and see if I can offer a bug fix.

Update: I have also experienced problems doing a course backup, with the same root cause, not enough disk space. When doing the course backup, the messages received are An error occurred while backing up course start and An error occurred while copying the zip file to the course directory. After looking at the code, it appears the Moodle error handling only looks for 4 types of IO errors. PHP 5 has 7 types, I believe it is the type 7 error that would cause this. Unfortunately the error trapping routine has a default handler for unknown error types.

Wednesday, July 30, 2008 4:24:06 PM (AUS Eastern Standard Time, UTC+10:00)       Comments [0]  Moodle | Programming | Webmaster

Using VisiFire to create a Silverlight chart of your visitors in dasBlog

Continuing from a previous post in which I added a Google Map of the visitors to my blog, this post will discuss how to take the same information and present it in a Visifire chart. Visifire is an open source Silverlight charting component which is very easy to use and well documented.

Figure 1 - An example of the Visifire Silverlight chart tracking visitors

This post will use the same component from the previous post, the VisitorsBox.ascx, and modify the code that retrieves the location of the visitor, and add some javascript to render the chart.

First, a JavaScript array is declared which will hold the location information received by the IPHost web site:

var cities=new Array();

This array will have items pushed into it once they are retrireved:

cities.push(new Array(location, addresses[x][1]));

Once again, the code that actually calls the HTML page with the location information is covered in the earlier post.

After the Google Map is populated, the Visifire chart is prepared. To create the Visifire chart we need to do the following:

• Add some HTML code into VisitorsBox.ascx
• Add the Visifire.js and Visifire.xap (SilverLight binary) files to the web server
• Dynamically create an XML file to feed to the Visifire.xap component
• in the JavaScript, create a new Visifre object
• Call the Visifire object's setDataXml method, passing the XML data in
• Call the Visifire objects render method, passing in the id of the HTML layer that the chart is to be rendered to.

The HTML code is straightforward, just a layer with an id attribute:

<div id="VisifireChart"></div>

The Visifire.js can be copied wherever you want, altho the scripts folder seems an obvious candidate. The Visifre.xap file can go in the bin folder.

To create the XML, I have just concatenated a large string. Where it requires the data, I loop thru the cities array and populate the relevant information. Note this is C# code writing out a JavaScript block. Here is the header part of the XML string:

function chart() {
var chartXmlString = ''
    +'<vc:Chart xmlns:vc=\'clr-namespace:Visifire.Charts;assembly=Visifire.Charts\''
+' Width=\'500\' Height=\'300\' BorderThickness=\'0\' Theme=\'Theme3\' ';
+'LightingEnabled=\'False\' ColorSet=\'Picasso\' AnimationType=\'Type5\'>'
+'<vc:Title Text=\'Visitors by location\'/>'
+'<vc:AxisX Title=\'blog.focas.net.au\'>'
+'</vc:AxisX>'
+'<vc:AxisY Title=\'Visitors\'>'
+'</vc:AxisY><vc:DataSeries Name = \'Series1\' RenderAs=\'Column\'>';

Now, to create the data points:

for (var x=0;x<cities.length;x++) {
    chartXmlString+= '<vc:DataPoint AxisLabel=\'' + cities[x][0] + '\' YValue=\'' + cities[x][1] + '\'/>';
}

And close the XML string off:

chartXmlString+='</vc:DataSeries>'
    +'</vc:Chart>';

The following JavaScript code creates the chart, loads the data and renders the chart:

var vChart =new Visifire('Visifire.xap',500,300);
vChart.setDataXml(chartXmlString);
vChart.render('VisifireChart');

And thats all there is to it. As in the previous example, if you take the client side approach, then you have cross site scripting issues. So the site will need to be added to a trusted zone. If this is not suitable then you will need to run the code on the server that retrieves the locations.

Download instructions

The instructions for installing this page are in the earlier post Please remember that In order to view the example for this, you would have to add http://blog.focas.net.au to your trusted zones in Internet Explorer.

visifire.zip (11.06 KB)

Thursday, June 26, 2008 11:15:30 PM (AUS Eastern Standard Time, UTC+10:00)       Comments [3]  dasBlog | Downloads | JavaScript | Webmaster | Silverlight

VSTO solution for context sensitive dynamic help in Word 2007

A project I developed called for context sensitive dynamic help inside of Microsoft Word, based on the current cursor location. The idea was that if the cursor is on an image, then help information would be presented regarding the business rules pertaining to images. I.E. what size can the image be, policies regarding alternative text for web publishing etc. To achieve this there were two components. One was a Quality Assurance component to check for conformance to business rules. The other was the dynamic help system. I present a whittled down version of the help system here. The Quality Assurance component will be discussed in a future entry. There is everything necessary to implement this as a full fledged context sensitive dynamic help system. The help information is presented in a custom task pane, and uses HTML.

Figure 1 -Word 2007 showing the Context sensitive dynamic help system

Architecture

This solution uses three main components:

• The VSTO addin
• A MessageBroker which receives events from the VSTO addin
• The Help UserControl which consumes events from the MesageBroker

This centralised messaging architecture is beneficial when other components are introduced such as the QualityAssurance component referred to earlier. The VSTO Addin wouldn't need to be changed, the MessageBroker, the QualityAssurance component would subscribe as listeners for the events raised by the MessageBroker. Another advantage of this approach is that security and logging would be easily implemented in the one location.

So, when the Addin loads, the WindowSelectionChanged event is subscribed to by the addin.

this.Application.WindowSelectionChange+= new Microsoft.Office.Interop.Word.ApplicationEvents4_WindowSelectionChangeEventHandler(
   Application_WindowSelectionChange);

It would be possible for the Help UserControl to subscribe directly, but by doing it this way we can define a few rules around what events get triggered, and also cache the current style so that the event is not consumed by every registered listener every time the cursor moves even though the style hasn't changed.

void Application_WindowSelectionChange(Microsoft.Office.Interop.Word.Selection Sel) {
     Word.Style style = (Word.Style)Sel.get_Style();
     if (! style.NameLocal.Equals(_currentStyle)) {
        _currentStyle = style.NameLocal;
        _messageBroker.Publish(typeof(MessageEventArgs), Sel.Paragraphs[1], style);
     }
}

Note in this code how it is important to retrieve an instance of the style object and cast it to type Word.Style. This is COM Interop in Word. It would be possible to write an extension method to get around this, but I don't think its necessary for this example.

The constructor for the Help UserControl accepts an instance of the MessageBroker. Note: I should probably make this class a Singleton.

The Help UserControl receives event notifications when the current style has changed. It loads a web browser control and loads an HTML page named after the style. if no page exists for that style, it will display a default help page. There is a tabbed interface, with an Infoand aHelp tab. The Help tab has not been implemented, the idea was to provide links to examples.

I have only included stripped down versions of HTML pages for three styles:

• Heading 1
• Heading 2
• Heading 3

It is simply a matter of creating a page witht the style name, and an extension of .html and dropping it in the help folder. The code that loads the page looks like this:

private void LoadHelpIntoBrowser(string currentStyle) {
    String helpFilePath = CreateHelpFilePath(currentStyle);
    if (File.Exists(helpFilePath)) {
       webInfo.Navigate(helpFilePath);
    } else {
       LoadHelpHomePage();
    }
}

private void LoadHelpHomePage() {
   string p = Path.Combine(Path.GetDirectoryName(
   System.Reflection.Assembly.GetExecutingAssembly().CodeBase), "..\\..\\help");
   String helpFilePath = CreateHelpFilePath("WordJester");
   webInfo.Navigate(helpFilePath);
}

private String CreateHelpFilePath(String style) {
   Uri baseUri =new Uri(
      Path.GetDirectoryName(System.Reflection.Assembly.GetExecutingAssembly().CodeBase));
   return String.Format("{0}{1}..{1}..{1}help{1}{2}.html", baseUri.LocalPath,
         Path.DirectorySeparatorChar, style);
}

The beauty of this approach is that you can add in as many styles as required, even non standard styles, and as long as a help file exists, it will be displayed. The file names will look like this:

Figure 2 - file names for the HYML pages based upon the word style name

Run the solution, and try creating some heading levels 1, 2 and 3, and move the cursor between them and check out the help in the custom task pane change. Let me know what you think of this application.

WordJester.zip (224.21 KB)

Thursday, June 19, 2008 10:37:37 PM (AUS Eastern Standard Time, UTC+10:00)       Comments [0]  Downloads | VSTO | Word 2007

dasBlog and the Validation of viewstate MAC failed error

I have been having the following error occuring a lot in the dasBlog eventlog.

System.Web.HttpException: Validation of viewstate MAC failed. If this application is hosted by a Web Farm or cluster, ensure that <machineKey> configuration specifies the same validationKey and validation algorithm. AutoGenerate cannot be used in a cluster. ---> System.Web.UI.ViewStateException: Invalid viewstate.

As this blog is not hosted on a web farm, I searched high and low, but hadn't found anyone else reporting the same problem. After some further investigation, I think I have found the cause. It is always on the CommentView.aspx page. I tried adding comments in various browsers, and that worked without a hitch. So in frustration I did what I should have done in the first place, i.e. completely read the error message. In most instances there was an interesting user agent string such as MRSPUTNIK 1, 5, 0, 19 SW. This is mentioned in a lot of forum entries as a harvester or scraper. I suspect it is trying to post a spam message on the comments page, but because it hasn't followed the normal process, it doesn't contain the view state that the page was accepting. This is almost certainly a bot, and this is why there is no viewstate to decrypt from a previous page. I also checked the IP address, and port number. The ip addresses had many forum entries. One example was 89.149.205.199. I found an interesting site called IPillion which traces IP addresses. So adding that IP address to the url gives http://www.ipillion.com/?ip=89.149.205.199 which reports this IP address as sending lots of spam comments.

So it seems that dasBlog is sort of preventing the spam comments, altho accidentally. I hope this entry helps others having the same issue, as I couldn't find anyone using dasBlog who had the same problem, or had traced it to spammers using bots.
 

Tuesday, June 17, 2008 11:07:26 PM (AUS Eastern Standard Time, UTC+10:00)       Comments [6]  Webmaster | dasBlog

Where are your blog visitors coming from in DasBlog?

See part II of this post here.

By combining the Google Maps API with an IP address host service such as HostIP.info, it is possible to present a map showing the current visitors to your blog, and to display them on a map. This implementation shows how to do this for dasBlog. The result will look something like this:

Notice that the shadow on the marker for the united States is darker, indicating more hits for that area. Each of the marker icons can be clicked to give location details and how many hits for that location.

Client side or server side location determination

The location of an IP Address can be determined at the server side, or at the client side. Each has its advantages and disadvantages. A brief but not complete list of pros and cons is:

• Client side
  • Pros
    • No delay in serving the page as the processing is done after the page has been serves to the client.
    • Easy to implement
  • Cons
    • Requires a call to a web service on a different domain which will cause security concerns
    • Does not allow for users to be able to modify what is their perceived location
• Server side
  • Pros
    • More control as you are not dependent on an external web service
    • It is easier to cache results if required
  • Cons
    • More difficult to implement, especially if you want to keep to the architecture of the software you are developing for
    • You must maintain and update your own databse of IP address ranges and locations

The client side solution

This solution relies on a client side solution, taking advantage of the excellent Prototype JavaScript Framework. Further down the track I might possibly change this to a server side solution, but as it is only for an administrative page it is not so important. Note that there are security concerns with doing this client side. It is necessary top make an AJAX call to an external webservice, and this could be malicious, especially if a site were hacked. In order to view the example for this, you would have to add http://blog.focas.net.au to your trusted zones in Internet Explorer. If you do this, please make sure you remove it immediately afterwards. Usually I would only add a trusted site if I were using it frequently. The example page can be viewed at Visitors.aspx (Please scroll down to the bottom of the page as there are styling issues. The page may need refreshing)

I created a server page called Visitors.aspx, and a user control called VisitorsBox.ascx. These are directly based upon Referrers.aspx and ReferrersBox.ascx in the standard DasBlog source code.

How it works

C# code in the page creates JavaScript which is then registered with the RegisterClientScriptBlock method of the ClientScript object. The javascript requires some dynamic variables to be created and this is why it is in the page. Also the way dasBlog is set-up seems to make it difficult to add JavaScript to just one page without using this method. Atwo dimensional array is created which contains unique IP addresses and a count of hits for those addresses. THis is then processed on the client side The JavaScript code to create the map follows, it is well documented in the Google Maps API documentation, so i won't say too much about it.
     
if (GBrowserIsCompatible()) {
     map = new GMap2(document.getElementById('map_canvas'));
     map.setCenter(new GLatLng(37.4419, -122.1419), 1);
     var mapTypeControl = new GMapTypeControl();
     var topRight = new GControlPosition(G_ANCHOR_TOP_RIGHT, new GSize(10,10));
     var bottomRight = new GControlPosition(G_ANCHOR_BOTTOM_RIGHT, new GSize(10,10));
     map.addControl(mapTypeControl, topRight);
     map.addControl(new GSmallMapControl());

Now we need to loop thru the IP address array and call the hostip.info page to get the city/state/country information. There are different calls that can be made to return HTML or XML. I used the HTML version as it is a smaller response, and easy to parse with regular expressions. If there is no information for an IP address it will return (Unknown). Thed regular expression trims them out. This database seems to be maintained by one person, so consider donating to them.

var myAjax;
      
 for (var x=0;x<addresses.length;x++) {
     myAjax = new Ajax.Request('http://api.hostip.info/get_html.php?ip=' + addresses[x][0] , 
     {
         method: 'get', 
         onSuccess: function(originalRequest) {
            var result=originalRequest.responseText;
            var tokens=result.split('\n');
            var count=addresses[x][1];
            var country=/\:[^\(]+/i.exec(tokens[0])[0].substring(1).strip();
            var city=/\:[^\(]+/i.exec(tokens[1])[0].substring(1).strip();
            var location=(city.length>0)? city + ',':'';
             location+=country;     
            varmessage=location + ': ' + addresses[x][1] + ' visitor';
            if (addresses[x][1] > 1) message+='s';
		

 Now we can call the GClientGeocoder object to turn the city/state/country information into a latitude/longtitude combination. The GClientGeocoder's getLatLng object expects a method to be passed in to handle a callback. In this code, it is used to create the marker on the map which can be clicked to view more information.

   var geocoder = new GClientGeocoder();
   geocoder.getLatLng(location,
   function(point) {  
      if (point) { 
         var marker = new GMarker(point);   
         map.addOverlay(marker);   
         GEvent.addListener(marker, 'click', function() {
         marker.openInfoWindow(message);
      });
   }
   });
		

This javascript is created in the C# code using a StringBuilder object, and is then added to the page using the RegisterClientScriptBlock method of the ClientScript object.

Page.ClientScript.RegisterClientScriptBlock(this.GetType(), "ipaddressdata", sb.ToString(), true);

Download

The download code contains the Visitors.aspx and Visitors.ascx page and component. These should be dropped into the newtelligence.DasBlog.Web folder of your dasBlog source code. Before going any further you will need to change the Google Maps key in VisitorsBox.ascx.cs file.

Page.ClientScript.RegisterClientScriptBlock(this.GetType(), "googlemaps",
    @"<script src='http://maps.google.com/maps?file=api&amp;v=2&amp;key=YOUR KEY GOES HERE'"+" type='text/javascript'></script>");
		

When you have compiled the code, drop those two files into the root of your website, and the newtelligence.DasBlog.Web.dll into the bin folder of your website. Remember to change the trust settings for your blog, and enjoy seeing who is visiting your site. Note that the Google Maps JavaScript and Prototype JavaScripts are served up from the Google servers so you don't need to download them.

Visitors.zip (10.57 KB)

Wednesday, June 11, 2008 11:53:11 PM (AUS Eastern Standard Time, UTC+10:00)       Comments [0]  Downloads | JavaScript | Webmaster

Web log file parsing with c#

In the past week I have been doing a lot of analysis on the web analytics reports that are being generated by AWStats (A good Open Source product, don't be put off by the arrogant download Firefox message) and another commercial product that we have an Enterprise license for. I won't name this product, as I am very disillusioned with it, its accuracy is very questionable.

In order to determine how accurate the statistics were, i chose one days log file for a reasonably busy website. This site is mainly a Monday to Friday website with peaks around 10 AM to 4 PM, and another small peak 6 PM to 8PM. Weekends see a lower usage, so I used a sunday as this would be the smallest log I could reasonably analyse. Thee log contained just over 100,000 entries. In order to analyse it, I used Visual Studion 2008, filtering using regular expressions. After a while I realised I would like to review these stats on an ongoing basis, so i wrote a small log parser. There is an excellent free parser available from Microsoft called Log Parser 2.2 however I wanted direct control, because I also want to create some graphs. My first version creates a beautiful graph using the excellent Open Source Silverlight data visualisation component called Visifire. I haven't included that code in this download as i wanted to clean it up and make it more re-usable and extensible.

This tool is not particularly robust as it is just a throwaway utility, but I tried to make it a bit extensible for future requirments. The log files I am parsing are Apache log files. So I designed a simple ILogFormatReader interface, and created an ApacheLogParser implementation. This doesn't populate all the fields, but it's easy to see how it works to finish off the implementation if more information is necessary.

The main issue with parsing log files is how they are separated. in this case the log file is space separated. Any fields that have spaces in them are surrounded by double quotes or square brackets. Another consideration is what to do if a log entry comes in with an invalid format. I didn't worry about this too much, as the web logs should work well if the first line is correct. if this code was to go into production then of course that would have to be refactored.

Not all log file are created equal

Each web server can be customised to record different informatin in the log files. Additionally proxy servers can modify what is sent thru to the web server, so it is important to check the format of any logfile before parsing. The log files I am parsing are using an Apache log file format. More information can be found on the Apache log files page and the Microsoft Log file formats in IIS (6.0) page.

For those interested, I found the statistics in AWStats very close to what I believe they should be. There were inaccuracies that I couldn't explain. To be fair, we are running an old version of AWStats so I assume that this might have been addressed in a newer version. The very expensive commercial application we use reports approximately twice the bandwidth that it should, and does not understand how to handle the JSESSIONID tacked on to the end of some JSP applications. It confuses the real resource with the session ID, and we get very iaccurate statistics as a result.

There is some minimalistic reporting to give an idea of how to use the parser. The output looks like thie image here.

There is a download included below. I would be interested to hear if anyone finds this useful. I hope to update this application at some stage to include some graphing output. I might use Visifire as mentioned earlier, or possibly Microsoft Excel. I think that is a better option as it would allow for richer maniupulation of the reporting after the log files have been parsed. Excel has such powerful features it would require good justification not to use it as a reporting mechanism. I think that AWStats doesn't do their reports justice by presenting them as they do, Everyones reporting requirements are different.

Download

The applicaition can be downloaded here:

LogParsing.zip (7 KB)

Tuesday, June 03, 2008 11:33:35 PM (AUS Eastern Standard Time, UTC+10:00)       Comments [0]  Downloads | Webmaster