I'm writing a CSS book

JavaScript archives Compatibility Thidrekssaga QuirksBlog Donations CSS book Politics Mobile About COH

contact

Testing mobile phones, or the masochist’s guide to gleeful self-flagellation

After seven months of mobile testing (as well as a wealth of inventive invective aimed at mobile devices) I think it’s time to share some of my experiences with others who are inclined to violent self-punishment.

Welcome to my world! Bring your whip, bring a first-aid kit, and let’s have some fun punishing ourselves.

Today we’ll discuss the process of testing mobile browsers. We will not talk about the test results or their interpretation, we’ll leave that gorefest for another time.

It takes more time

The first and most important rule is that testing mobile phones will take more time than you think, even if you assume it’ll take more time than you think.

Tip: There is no such thing as a quick phone test. Always schedule at least 15 minutes, even if you just want to test one thing on one phone.

Deactivation

The first problem is that mobile phones are assumed to run on their cell power and therefore automatically deactivate after a minute or so of inactivity. This means you’ll have to switch the phone on before you can test anything. This is not a huge problem, but still, if you test many phones simultaneously the waiting time may become significant.

Tip: Set the timeout after which the phone goes inactive to the maximum value.

Sure, that’ll drain the battery faster, but so what? You’re testing at home or in your office anyway, where you can always plug the phone into AC power.

Deactivation is especially a problem with test suites such as the Browserscope. These tests take a while to run, and chances are your phone will go to sleep in the middle of such a test. So you can’t auto-run them; you have to stay nearby to press a button every minute or so.

Opening the browser, typing the URL

The second problem with quick tests is that it’s sometimes pretty tricky to start up the browser. Many phones (iPhone, Android, Nokia, Blackberry) have a browser icon on their home screen, but others (Windows Mobile) don’t.

Once you’ve opened the browser you have to go to the option that allows you to type in a URL. Sometimes these options are exceptionally well hidden.

Then there’s the problem of typing in the URL to your test pages. This can be really time consuming on a phone with a numeric keypad, and even on phones that have a (hardware or software) keyboard it’s still not what you’d call easy. Believe me, you don’t want to laboriously type a long URL into seven different phone interfaces.

Tip: Create a portal page that contains links to all your tests and store the URL of this page as a bookmark on every phone. Make sure this URL is short. (I use http://quirksmode.org/m.)

If you add new tests you can update this portal page and don’t have to worry about typing in URLs.

You do have to worry about caching, though.

Caching

All mobile browsers have their own caching policy, which ranges from non-existent (Safari iPhone) to extremely aggressive (S60 WebKit).

Tip: once an S60 WebKit has your test page in its caching clutches it will never release it. If you modify a test page you must manually flush the cache (in the Options menu). Every single time.

Other browsers generally handle page modifications better.

Connections

Then there’s the problem of connections. On bad days, just getting the phones to connect to anything outside themselves can take a quarter to a third of my time.

Obviously, wifi is by far the fastest and cheapest way to connect to the Internet. Many phones support wifi, but some (notably Blackberry) don’t.

Wifi

Windows Mobile is by far the most annoying wifi-capable OS. When a Windows Mobile phone goes inactive it’ll shut down its wifi, but when you activate it again, it usually does not automatically restart its wifi. (This problem is less severe on Win Mob 6.5, by the way.)

Tip: If a Windows Mobile phone doesn’t have a wifi connection after activation, switch to the communications manager, turn wifi off and then on. This usually helps.

The Nokia S60 WebKit browser always tries to connect to the last wifi network you were connected to and gives you no way of switching to another network. In my case that’s really annoying because it has to alternate between my home network and Vodafone’s network in Düsseldorf.

Tip: If you alternate wifi networks on the Nokia, click on Search for Network on the home screen, select the network you want, and then click Start Browsing in the options menu.

Also, whichever browser you start up, a Nokia will always ask you twice if you’re really sure you want to commence “off-line” browsing (i.e. browsing without a SIM card). And when you switch browsers it’s sure to ask you the same question again. Twice. It’s almost as if Nokia is surprised that you want to connect to the Internet.

SIM card

The fact that some phones don’t support wifi means you need a SIM card. The point is that you’ll likely have one single SIM card available, and several phones that need one.

Tip: Write down the PIN code of your SIM card on a piece of paper. Do not under any circumstance throw away that piece of paper. (A true story from the QuirksMode labs.)

Therefore you need to switch your SIM card from phone to phone, and the problem is that you have to remove the battery in order to do so. Thus, once you’ve inserted the SIM card into a phone you have to wait for it to start up.

This is a problem especially on Blackberry, which takes at least 4 to 5 minutes to restart. Other phones are relatively fast, but “relatively” still means about 1 to 2 minutes of waiting time.

Tip: Decide on the order you’re going to test the phones in beforehand. Make sure that you alternate between wifi-capable and non-wifi phones. When you finish with a non-wifi phone, switch the SIM card immediately to the next non-wifi phone, and test a wifi phone while the non-wifi phone starts up.

Bluetooth (or Why Apple and Google are Evil)

Since I’m doing research into the interoperability of W3C Widgets I often want to transfer test widgets via Bluetooth. This is not as easy as it seems.

The iPhone and Android do not support Bluetooth in any meaningful way. It’s totally impossible to connect to another phone. I expected no less from Apple, but Google’s going over to the Dark Side caught me by surprise.

Windows Mobile phones have a ridiculously complicated Bluetooth menu that I can’t make heads or tails of. Entering a phone name that other phones will see in their Bluetooth menu, which is a relatively smooth process on the Nokia, seems to be totally impossible on Windows Mobile phones. (If you know how to do it, please leave a comment.)

Fortunately Windows Mobile phones allow you to mount the file system of another phone via Bluetooth. Essentially, this allows you to browse through the other phone in the Windows Explorer. The only minor annoyance is that you have to OK every directory switch on the source phone.

Tip: Some phones will ask for a Bluetooth password after you’ve OK’d the connection. Use the same simple password for all phones; it’ll save you a lot of hassle. I always use “1.”

Content adaptation

Tip: If you use a SIM card and your phone suddenly starts up an old Opera Mini 2 or even weirder browsers instead of the default browser you were expecting, and you see all kinds of menu bars you didn’t put in your test page, you have to turn off the SIM card’s content adaptation setting (which is never called the content adaptation setting).

Carrier content adaptation is by far the most Evil thing I’ve encountered in the mobile space — and that includes everything Apple has done or will do.

Basically, content adaptation means that the SIM card’s carrier has in its infinite wisdom decided that your phone’s browser is unable to properly access the Internet. Therefore it starts up an Opera Mini-like browser that renders the page on a server and then sends it on to your phone; and it thoughtfully includes extra menu bars and stuff you never asked for and the page author doesn’t know about.

In theory content adaptation could be useful, but in practice the carriers have no fucking clue what they’re doing. Vodafone gave me a customized Opera Mini 2 instead of the NetFront 3.2 I was expecting. Now NetFront is not a particularly good browser, but it can still handle web sites. I didn’t need an Opera Mini 2 and extra menu bars.

Finding out that this browser announces itself client-side as Opera Mini 2 but server-side as NetFront 3.2 was definitely the most chilling moment I’ve had in my entire career.

Content adaptation is a SIM card setting, and you can switch it off for your SIM card (at least, you can in the Vodafone network). One of the Vodafone guys switched it for me, and I can’t really remember what he did, but somewhere deep in some menu structure there is a toggle.

Testing

When I run compatibility tests on desktop browsers, I run the same test in all of them, compare the results, and write a compatibility table entry. As far as I’m concerned this is the best way of working, since it allows me to judge compatibility problems one by one and compare the rendering in various browsers.

Testing order

Sadly this way of working is impossible on mobile. Once you’ve got one phone to load the page and do the test, the screen of the other phones has gone inactive again and you have to reactivate them. Although it doesn’t take that much time, it’s hellishly complicated to keep all test phones active and do something meaningful simultaneously.

Tip: Make sure you do not need to compare more than two phones side by side.

Therefore, I do all tests on one phone, then switch to another and repeat all tests. This is less optimal, because it doesn’t allow me to easily compare two phones’ rendering or execution of my tests, but it’s by far the most time-saving way of working.

Where in my desktop browser tests I can jot down a vague hint to myself and retest the browser in a matter of seconds if I don’t understand my own notes, during my mobile tests that’s impossible because it takes at least a minute or so to retest a phone I’ve put aside earlier — and possibly much longer.

Tip: Keep really detailed notes during the entire testing process. Make sure that you write down something that somebody else can understand and use before moving to the next test (or phone).

Also remember the earlier tip: make sure to alternate between wifi-enabled and non-wifi phones. I usually create a row of phones and test them from left to right, switching SIM cards as soon as I’m ready with one non-wifi phone.

Mark the test start

On some phones (notably Windows Mobile) it’s not very easy to determine whether you’ve clicked the “Start test” link and the test is running, or whether you’ve missed it by a few pixels and the phone is doing nothing.

Tip: make sure you change something on screen when the test starts to run — a little “test running” text is enough. Thus you’ll always be certain that you have in fact clicked on the link and the test is in fact running.

Testing non-default browsers

If you’re like me you’re not content with testing a system’s default browser, but also want to test alternatives that end users may download; notably Opera Mini.

Tip: Pick a wifi-capable S60 or Windows Mobile device you particularly like and install all non-default browsers on it.

Most modern non-default browsers run on J2ME systems (S60, Windows Mobile, Blackberry), and putting them all on one phone you feel comfortable with greatly helps your testing process. I put all minor browsers on the Nokia E71, which is my personal favourite.

Screenshots

Tip: If you find a good, fast way of making screenshots of mobile phones, please write a blog post about it to help other people.

Making screenshots of mobile phones is pretty hard work, and I haven’t yet found a process that satisfies me. Part of the problem is that I don’t have a good camera; maybe it’s time to buy one.

Vodafone has bought a shiny new Elmo P30s projector/camera, but I’m sad to report that its screenshot capabilities are modest at best. The process is pretty straightforward, but we weren’t able to get rid of an annoying moiré effect that adds a grid of black lines to most screenshots. See my Yahoo! mobile browser presentation for some examples of mobile screenshots taken with the Elmo.

Some mobile OSs (I’m certain of iPhone and S60) have a screenshot application that you can use. Still, this application only takes a shot of the actual screen, and not of the entire mobile device. This may matter sometimes, especially when you try to explain the difference between a touchscreen and a four-way navigation phone. In that case, it’s best to show the phone hardware interface in the screenshot, too.

I hope to be able to tell you more about mobile screenshots later on.

Testing mobile phones

Armed with this knowledge we’re finally ready to start with the actual testing. We’re going to find all kinds of exciting new browser incompatibilities that just don’t exist on desktop computers. It’s going to be such fun!

If you feel your life is too painless and smooth right now I recommend testing some mobile phones.

Comments

Comments are closed.