Introduction to W3C Widgets

As I said before, I’m currently working for Vodafone on mobile browser compatibility and W3C Widgets. I’ve discussed some mobile browser problems, and you can look over my shoulder while I’m at work dissecting their odd behaviours. If you want the latest scoops on my mobile adventures, you can follow me on Twitter.

The time has come to talk about the W3C Widgets part of my job. Exactly what is a widget, how do you create one, why would you want to, and which systems support them?

Personally I firmly believe that widgets are the future of the mobile web. They are easy to create, they’re based on open standards, they save the end user quite a bit of network traffic, and many people around the world already know how to create them.

In contrast to other recent publications about widgets, I’ll tell you the whole story — or rather, a condensed version thereof.

What is a widget?

Essentially, a widget is a local HTML/CSS/JavaScript web application. A mobile phone user downloads a widget once, and from that moment on he has a web application stored locally on his mobile phone.

Although currently widgets are mostly used for relatively limited functions such as clocks, RSS readers, or Twitter or Flickr clients, as well as for games, there’s no theoretical reason why they couldn’t contain a complete, complicated, JavaScript-heavy web application; for instance a mobile-optimised spreadsheet application.

The beauty of the widget model is that, even if this spreadsheet application needs 200K of custom JavaScript plus a few libraries, the end user has to download these scripts only once instead of every time he starts up the app, as in the traditional web application model. After the initial download the entire application is installed locally and can be fired up from the local files.

Many web applications will want to display fresh data, and therefore most widgets will fire an Ajax request as soon as they’re started up. For that they need a network connection and some data traffic.

Nonetheless, the fact that the application itself doesn’t have to be downloaded every time the user starts it up diminishes the network traffic by a lot; and that’s important on mobile phones with their unreliable and sometimes crawlingly slow connections.

Unique selling point

In itself the widget concept is no different from the countless application models currently being set up for the iPhone, Android, Blackberry, Windows Mobile, and many more systems. These, too, will put the core files on your phone and use the network connection only for data traffic.

All proprietary systems, however, require programmers to train themselves in new skills and techniques that are not interoperable. Widgets, on the other hand, use the open web technologies that hundreds of thousands of developers all over the world are already familiar with.

Better still, widgets will eventually conform to the “write once, run anywhere” principle; though right now that’s not yet the case.

Personally I firmly believe that widgets are the future of the mobile web. They are easy to create, they’re based on open standards, they save the end user quite a bit of network traffic, and many people around the world already know how to create them.

Technical overview

Widgets are nothing more than zipped websites. You create an HTML file, add as many CSS, JavaScript and image files as you need, zip the lot, change the extension to .wgt and it works. Just like that.

Well, almost just like that. The rest of this entry explains the “almost.”

Specification

Widgets are specified by the W3C Widgets specification. Currently this spec is still a work in (slow) progress, as so many W3C specifications are, but the outline is pretty clear.

The widget environment I’m working with has been created by Vodafone in close cooperation with Opera. Because they wanted to work according to a standard but the specification was (and is) still in draft stage, at a certain point they decided to freeze the then-current version and implement it. Subsequently it was slightly modified to solve some urgent problems.

All technical details I’m going to give come from this Opera/Vodafone implementation, which does not quite match the current W3C draft. Everything works, though — I tested it personally.

(Once the specification stabilises Opera and Vodafone plan to implement it, and that will probably mean already created widgets will experience minor breakage. We’re planning to release a library that works around these issues for you.)

Widget SDK

Unfortunately there’s a zipping issue on Mac. I’m not sure of the details, but if you just use the default zip on Mac, your widget will not work.

In order to solve this problem and several others, Vodafone has released a Widget SDK. Although the site doesn’t state so clearly, it’s in beta right now and a final release is expected later this year. You should take a look at it if you’re serious about creating widgets.

It contains a special version of the Vodafone Widget Manager that allows you to debug your widgets through a Bluetooth connection with Opera Dragonfly. The procedure is sometimes complicated; see Debugging Mobile Widgets with Opera Dragonfly for the technical details.

config.xml

A widget requires a config.xml file that specifies some configuration information. Here’s a sample config file:

<widget id="http://quirksmode.org/widget" dockable="true">
  <widgetname>Test widget</widgetname>
  <icon>pix/myIcon.gif</icon>
  <width>200</width>
  <height>200</height>
  <security>
    <access>
      <host>quirksmode.org</host>
    </access>
  </security>
</widget>

Most of this stuff is pretty simple. <width> and <height>, unsurprisingly, define the width and height of the widget in pixels.

These are start values; if you want you can later on change the widget size by the normal resizeTo() and resizeBy() methods. The widget cannot in any case become larger than the screen.

The <icon> tag specifies the icon that the user sees when the widget is turned off; and this is of course a local file.

The <widgetname> tag contains the name that will be shown with the widget icon. It’s advisable to use short names, because there’s only about 60 pixels of space to show the name when the widget is docked. Some phones auto-scroll longer texts when the user focuses on the icon, but that’s not an optimal means of communication.

The id attribute is required if you want to be able to update your widget. I’ll get back to the updating process in a later post; for now it’s sufficient to say that an id must be a unique URI.

As to the <host> element, it specifies with which host(s) the widget is allowed to communicate via Ajax requests. Widgets don’t really have a same-source policy; they’d be worthless if they did because they must be able to download data from anywhere. However, the widget creator is required to specify which domains the widget may access.

(Unfortunately Opera unexpectedly changed the security policy in Opera 10 Desktop. You need the syntax shown above for Opera 9 and the Widget Manager, but a different one for Opera 10.)

Docked widgets

Widgets have a docked state that’s pretty similar to minimised applications on your desktop computer, in that scripts may continue to run in a docked widget; for instance to silently update an RSS reader with new data.

However, you have to explicitly allow scripts to continue running by adding the dockable="true" attribute to the <widget> tag. If it’s not there, the widget assumes it may not run any scripts while docked.

While a widget is docked, it is still visible to the user as an icon, although the viewport now becomes about 60x60 pixels (and remains so until the user undocks the widget).

If you wish you can present the user with a special interface while the widget is docked; for instance a special icon. In order to do that you need to set a widgetmodechange event handler that checks if the current widget.widgetMode equals docked and if so hides the normal interface and shows a special, 60x60 optimised one.

The widget object and external links

In a widget environment an extra widget object is added to the JavaScript global object to contain these useful widget functionalities such as the widgetmodechange event and the widgetMode property. You can find a complete overview in the widget object compatibility table.

Finally, normal hyperlinks open not in the widget itself, but in the default S60 WebKit browser (on mobile phones) or in the regular Opera (on desktop). I do not understand why this should be the case. A widget environment is capable of handling HTTP requests and interpreting HTML, CSS, and JavaScript, after all.

Support

Although I firmly believe support for W3C widgets will become better and better as time progresses, right now it’s still in its infancy.

Opera is a major defender of the widget concept, so it comes as no surprise that Opera Desktop supports it. If you’ve created a .wgt file on your computer, double-clicking it starts up Opera, which shows the widget in a special widget mode. That’s useful for testing.

Vodafone Widget Manager

On mobile phones the situation is murkier. The Vodafone Widget Manager, that runs only on Symbian S60 phones (basically Nokia), supports W3C widgets, but few other applications or phones do.

The Vodafone Widget Manager (surprise!) has been created by Vodafone in order to manage widgets. It contains a system for installing them, as well as a complete Opera Mobile runtime engine that is used to actually interpret the HTML, CSS, and JavaScript.

There are some slight differences between a regular Opera Mobile and the Opera Widget Runtime. In my tables this special Opera is noted as “Opera VF WM” (for Vodafone Widget Manager).

Opera is a pretty good browser, so you can be pretty certain that your widgets will run pretty well as long as you use proper HTML, CSS, and JavaScript. On the other hand, this is a new, untested technology, so nasty surprises will definitely turn up at the worst possible moment in your development cycle.

Opera/T-Mobile Widget Manager

Opera announced just now that it has built a similar system for T-Mobile. Because all the screenshots feature Windows Mobile interfaces, we kind of assume it runs (only?) on that OS.

Confusingly, when I search for “widgets” on the T-Mobile site I get three pages about other widget systems on the German site, whilt the UK and US sites each have one unrelated match.

Interestingly, this widget manager seems to contain a device API. A device API is a JavaScript interface meant for accessing phone functionality such as the camera, the calendar, the contact list, and so on. It’s obvious that device APIs are going to be extremely important for creating widgets that tie in more closely with mobile phones.

As far as I can see now the Opera/T-Mobile device API is proprietary (and possibly restricted to Windows Mobile). That’s not entirely unexpected; currently the BONDI specification is supposed to formulate a standard for device APIs, but it’s not ready yet and unreadable for non-Java programmers anyway.

Despite this touch of proprietariness I’m very happy with the Opera/T-Mobile initiative. The more companies support widgets, the sooner they’ll go mainstream.

Nokia Web Runtime Widgets

I think Vodafone widgets can be ported to Web Runtime Widgets, the native Nokia S60 widget environment. These, too, are HTML/CSS/JavaScript widgets, although they run in S60 WebKit and need an info.plist XML file instead of a config.xml.

As far as I can see, making your widget Nokia-compatible entails nothing more than adding this info.plist to it, and making sure your widget runs well in S60 WebKit. I haven’t tried this for myself, though, so there may be unforeseen circumstances that complicate the matter.

By the way, the “XML” in the plist is weird. Somebody tried (and failed) to port name/value pairs to XML. The schema is based on Apple’s Dashboard Widgets, so if you know those you’ve got the advantage over me here.

Palm Pre?

Maybe the Palm Pre, which has made Web technologies the very cornerstone of its webOS operating system, will support them, but there’s so little data available that I can’t really tell, except that it would be a logical extension of their plans. The Palm Pre uses a WebKit branch as default browser, by the way.

Other than that I’m not sure where we can easily deploy our widgets right now. I do advise you, however, to test your widgets in a WebKit (Safari or Chrome being the most obvious candidates) in addition to Opera. It will save you some hassle later on.

Our To Do list

Ideally, in a few years, I, using an Android, would be able to transfer a widget to a friend’s S60 Nokia and another friend’s Windows Mobile HTC via Bluetooth (or whatever) and They’d Just Work without me or them having to worry about compatibility.

Wouldn’t that be wonderful? Wouldn’t that be absolutely astoundingly interoperable?

Besides, widgets will also work on desktop computers. As I said, Opera already supports them, and it’s only logical that other browsers will eventually do so, too.

Still, this brave new world won’t give birth to itself. We, as web developers embracing open standards, have our work cut out for us. We have to evangelise standard-compliant W3C Widgets. If you see me speak somewhere during this year, chances are I’ll pontificate on exactly that subject.

However, we should be careful to stress that we want widgets not instead of, but in addition to the proprietary systems mobile phone vendors are setting up right now.

These systems have their place, and they might conceivably be a better environment for certain types of applications. Besides, everybody is allowed to innovate and try out new platforms and ideas, as long as they also support the standard.

It will not be easy. Neither will it be quick. But open technologies, and especially open web technologies, are in an upswing throughout the world, while closed systems will gain a hard core of enthousiastic developers, but not the tidal-wave effect their creators so desperately want.

If I were a Marxist I’d say our victory is unavoidable, but since I’m not quite that orthodox I’ll say that it’s likely we’ll succeed.

This article has been translated into Russian.

This is the blog of Peter-Paul Koch, mobile platform strategist, consultant, and trainer. You can also follow him on Twitter.
Atom RSS

I’m around at the following conferences:

(Data from Lanyrd)

Categories:

Monthlies:

Comments

Comments are closed.

1 Posted by Harald Kirschner on 20 April 2009 | Permalink

Great roundup, thanks a lot for sharing your research. I only developed simple Nokia WRT widgets, never really looked left and right. Theoretically simple modifications seem to be enough to make cross-device widgets ... surprise, browsers complicate it again!

2 Posted by Tim Tepaße on 20 April 2009 | Permalink

The property list metaformat comes from Nextstep. It exists in ASCII-Format like JSON, was then ported to XML, then to a binary XML:
http://developer.apple.com/iphone/library/documentation/Cocoa/Conceptual/PropertyLists/UnderstandXMLPlist/UnderstandXMLPlist.html#//apple_ref/doc/uid/10000048i-CH6-SW1

There's a reference for Dashboard keys:
http://developer.apple.com/documentation/AppleApplications/Reference/Dashboard_Ref/DashboardPlist/DashboardPlist.html

3 Posted by Alec on 20 April 2009 | Permalink

Thanks for the post. I hadn't realized there was an open standard already out there. Following you on twitter.

4 Posted by Ryan Cannon on 20 April 2009 | Permalink

Pretty fascinating. Adobe Air seems like a really great candidate for bringing the widget framework to the desktop. It might not be horribly tough to create an Air App that can render w3c widgets. Or, one might want to petition Adobe to add a compatibility layer between Air and widgets.

http://www.adobe.com/products/air/

5 Posted by Dave Johnson on 21 April 2009 | Permalink

There is also an open source project called PhoneGap (http://www.phonegap.com) that exposes many of the native phone APIs to a browser sandbox allowing cross plaform native application development using open web standards.

6 Posted by Jeff Sonstein on 22 April 2009 | Permalink

the link for phonegap has a spurious paren in it, and it can actually be reached here:

http://www.phonegap.com/

7 Posted by Kit Grose on 23 April 2009 | Permalink

.plist files are Property List files, and come courtesy of NeXT (and so Mac OS X by proxy).

You can thank someone at Apple for the horrible XML structure they invented for these things; as far as I can tell, it's to allow very permissive key names, but I don't like sibling nodes to be both keys and values.

If you're on a Mac, editing plists is greatly helped along by Property List Editor which comes with the free Developer Tools. Windows users can use plist Editor for Windows from http://www.iPodRobot.com/download.htm

The structure you define closely matches Dashboard widgets in OS X (which, IIRC, are also supported in the Opera Widget Engine), so it's possible you could use any of the great Dashboard widget IDEs to build those (Dashcode is the Apple one).

8 Posted by JJ Halans on 27 April 2009 | Permalink

Great overview!
Though I'm looking into more info on widgets on TV. In order to distinguish themselves, TV manufacturers start embedding internet connected widgets. Like Samsung with so called Internet@TV widgets, which are actually Yahoo! widgets.
I also read about a prototype with Chumby (Flash) widgets on TV. So next to mobile widgets, I think widgets hold a great future on TV too!

9 Posted by Mark on 3 May 2009 | Permalink

This maybe isn't quite the right post to add this comment to, but do you have something similar to http://htmldog.com/test/handheld.html in your test pages?

10 Posted by Bob Mitchell on 3 May 2009 | Permalink

No idea that this was in the works. It's something that I'll be following very closely as it looks to have great potential.

One day, with proper standards-supporting browsers we may be at the point where cross-platform write-once run-anywhere actually works, ironically not using the language originally designed for such things.

Time to have a play I think! :-)

11 Posted by Andrew on 24 May 2009 | Permalink

Thanks for the great post!
I translated it into Russian, thinking it would be great to propagate the information. The backlinks are, of course, provided. The article itself is here http://interpretor.ru/w3c_widgets/

12 Posted by Alper Çugun on 12 June 2009 | Permalink

I was at the same event and I was somewhat baffled by the lack of communication between the various widget standards.
See: http://alper.nl/dingen/2009/05/mobile-widget-camp/

The Vodafone platform is a third tier play by Opera which normally can't easily get on the cell phones. It makes sense that a cell phone manufacturer like Nokia would have their own standard.

What was missing for me was any mention of OpenSocial which is in essence the same thing: a window to render HTML content with specific JavaScript APIs based on host platform capabilitiy. It doesn't matter much if the host platform is a social network or a cell phone.

OpenSocial is a standard which has a lot of support and an open implementation going for it. I'm surprised nobody is talking with/about them.

You're right that it's not too difficult to port any one widget to another runtime, but having to maintain all those final 20% of adaptations will kill you.

13 Posted by Tom on 8 July 2009 | Permalink

I just wanted to add that the from what I see now, Palm Pre is beginning of the mobile adventure for these widgets. It's a little messy of a process, but so far it looks like anyone can easily port almost any Oprea widget to the Pre. Of course you'd want to add their custom tags to support rotating the phone, etc. But this approach of the widget apps on there phone is going to really explode there store quickly. Getting the SDK and following the install instructions I've ported a few apps... IP/copyright protecting is going to be an issue.